boom boom

* fix USE_CUDA, add pipeline

* USE_CUDA fix

* recode SinusoidalPositionalEmbedding into nn.Embedding subclass

was needed for torchscript to work - this is now part of the state_dict, so will have to remove these keys during save_pretrained

* back out (ci debug)

* restore

* slow last?

* facilitate not saving certain keys and test

* remove no longer used keys

* style

* fix logging import

* cleanup

* Update src/transformers/modeling_utils.py

Co-authored-by: Sam Shleifer <sshleifer@gmail.com>

* fix bug in max_positional_embeddings

* rename keys to keys_to_never_save per suggestion, improve the setup

* Update src/transformers/modeling_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sam Shleifer <sshleifer@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

README.md

@@ -48,7 +48,7 @@ Here are a few examples:
 
 ## Quick tour
 
-To immediately use a model on a given text, we provide the `pipeline` API. Pipelines group together a pretrained model with the preprocessing that was used during that model training. Here is how to quickly use a pipeline to classify postivive versus negative texts 
+To immediately use a model on a given text, we provide the `pipeline` API. Pipelines group together a pretrained model with the preprocessing that was used during that model training. Here is how to quickly use a pipeline to classify positive versus negative texts 
 
 ```python
 >>> from transformers import pipeline
@@ -83,7 +83,7 @@ To download and use any of the pretrained models on your given task, you just ne
 >>> from transformers import AutoTokenizer, AutoModel
 
 >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
->>> model = AutoModel.from_pretrained("bert_base_uncased")
+>>> model = AutoModel.from_pretrained("bert-base-uncased")
 
 >>> inputs = tokenizer("Hello world!", return_tensors="pt")
 >>> outputs = model(**inputs)
@@ -93,7 +93,7 @@ or for TensorFlow:
 >>> from transformers import AutoTokenizer, TFAutoModel
 
 >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
->>> model = TFAutoModel.from_pretrained("bert_base_uncased")
+>>> model = TFAutoModel.from_pretrained("bert-base-uncased")
 
 >>> inputs = tokenizer("Hello world!", return_tensors="tf")
 >>> outputs = model(**inputs)

docs/source/index.rst

@@ -3,9 +3,9 @@ Transformers
 
 State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0.
 
-🤗 Transformers (formerly known as `pytorch-transformers` and `pytorch-pretrained-bert`) provides general-purpose 
-architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet...) for Natural Language Understanding (NLU) and Natural 
-Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between 
+🤗 Transformers (formerly known as `pytorch-transformers` and `pytorch-pretrained-bert`) provides general-purpose
+architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet...) for Natural Language Understanding (NLU) and Natural
+Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between
 TensorFlow 2.0 and PyTorch.
 
 This is the documentation of our repository `transformers <https://github.com/huggingface/transformers>`_.
@@ -127,7 +127,7 @@ conversion utilities for the following models:
 23. `Pegasus <https://github.com/google-research/pegasus>`_ (from Google) released with the paper `PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
     <https://arxiv.org/abs/1912.08777>`_ by Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu.
 24. `MBart <https://github.com/pytorch/fairseq/tree/master/examples/mbart>`_ (from Facebook) released with the paper  `Multilingual Denoising Pre-training for Neural Machine Translation <https://arxiv.org/abs/2001.08210>`_ by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov,
-    Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.  
+    Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.
 25. `LXMERT <https://github.com/airsplay/lxmert>`_ (from UNC Chapel Hill) released with the paper `LXMERT: Learning
     Cross-Modality Encoder Representations from Transformers for Open-Domain Question
     Answering <https://arxiv.org/abs/1908.07490>`_ by Hao Tan and Mohit Bansal.
@@ -223,6 +223,7 @@ conversion utilities for the following models:
     model_doc/dpr
     model_doc/pegasus
     model_doc/mbart
+    model_doc/fsmt
     model_doc/funnel
     model_doc/lxmert
     model_doc/bertgeneration

docs/source/model_doc/fsmt.rst

@@ -0,0 +1,49 @@
+FSMT
+----------------------------------------------------
+**DISCLAIMER:** If you see something strange,
+file a `Github Issue <https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title>`__ and assign
+@stas00.
+
+Overview
+~~~~~~~~~~~~~~~~~~~~~
+
+FSMT (FairSeq MachineTranslation) models were introduced in "Facebook FAIR's WMT19 News Translation Task Submission" <this paper <https://arxiv.org/abs/1907.06616>__ by Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, Sergey Edunov.
+
+The abstract of the paper is the following:
+
+    This paper describes Facebook FAIR's submission to the WMT19 shared news translation task. We participate in two language pairs and four language directions, English <-> German and English <-> Russian. Following our submission from last year, our baseline systems are large BPE-based transformer models trained with the Fairseq sequence modeling toolkit which rely on sampled back-translations. This year we experiment with different bitext data filtering schemes, as well as with adding filtered back-translated data. We also ensemble and fine-tune our models on domain-specific data, then decode using noisy channel model reranking. Our submissions are ranked first in all four directions of the human evaluation campaign. On En->De, our system significantly outperforms other systems as well as human translations. This system improves upon our WMT'18 submission by 4.5 BLEU points.
+
+The original code can be found here <https://github.com/pytorch/fairseq/tree/master/examples/wmt19>__.
+
+Implementation Notes
+~~~~~~~~~~~~~~~~~~~~
+
+- FSMT uses source and target vocab pair, that aren't combined into one. It doesn't share embed tokens either. Its tokenizer is very similar to `XLMTokenizer` and the main model is derived from `BartModel`.
+
+
+FSMTForConditionalGeneration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.FSMTForConditionalGeneration
+    :members: forward
+
+
+FSMTConfig
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.FSMTConfig
+    :members:
+
+
+FSMTTokenizer
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.FSMTTokenizer
+    :members:
+
+
+FSMTModel
+~~~~~~~~~~~~~
+
+.. autoclass:: transformers.FSMTModel
+    :members: forward

docs/source/testing.rst

@@ -756,7 +756,14 @@ To run such tests set ``RUN_SLOW=1`` env var, e.g.:
 .. code-block:: bash
 
     RUN_SLOW=1 pytest tests
+    
+It's important that the decorator ``@slow`` appears last in the stack of decorators, as some decorators like ``parametrized`` may interfere with its normal functioning. Here is an example of the correct usage:
 
+.. code-block:: python
+
+    @parameterized.expand(...)
+    @slow
+    def test_integration_foo():
 
 Testing the stdout/stderr output
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

examples/lightning_base.py

@@ -119,7 +119,7 @@ class BaseTransformer(pl.LightningModule):
     def get_lr_scheduler(self):
         get_schedule_func = arg_to_scheduler[self.hparams.lr_scheduler]
         scheduler = get_schedule_func(
-            self.opt, num_warmup_steps=self.hparams.warmup_steps, num_training_steps=self.total_steps
+            self.opt, num_warmup_steps=self.hparams.warmup_steps, num_training_steps=self.total_steps()
         )
         scheduler = {"scheduler": scheduler, "interval": "step", "frequency": 1}
         return scheduler
@@ -159,19 +159,20 @@ class BaseTransformer(pl.LightningModule):
     def test_epoch_end(self, outputs):
         return self.validation_end(outputs)
 
-    @property
     def total_steps(self) -> int:
         """The number of total training steps that will be run. Used for lr scheduler purposes."""
         num_devices = max(1, self.hparams.gpus)  # TODO: consider num_tpu_cores
         effective_batch_size = self.hparams.train_batch_size * self.hparams.accumulate_grad_batches * num_devices
-        dataset_size = len(self.train_loader.dataset)
-        return (dataset_size / effective_batch_size) * self.hparams.max_epochs
+        return (self.dataset_size / effective_batch_size) * self.hparams.max_epochs
 
     def setup(self, mode):
-        if mode == "fit":
+        if mode == "test":
+            self.dataset_size = len(self.test_dataloader().dataset)
+        else:
             self.train_loader = self.get_dataloader("train", self.hparams.train_batch_size, shuffle=True)
+            self.dataset_size = len(self.train_loader.dataset)
 
-    def get_dataloader(self, type_path, batch_size, shuffle=False):
+    def get_dataloader(self, type_path: str, batch_size: int, shuffle: bool = False):
         raise NotImplementedError("You must implement this for your task")
 
     def train_dataloader(self):

examples/requirements.txt

@@ -5,7 +5,7 @@ psutil
 sacrebleu
 rouge-score
 tensorflow_datasets
-pytorch-lightning==0.8.5
+pytorch-lightning==0.9.0
 matplotlib
 git-python==1.0.3
 faiss

examples/seq2seq/README.md

@@ -227,7 +227,9 @@ python run_eval.py sshleifer/distilbart-cnn-12-6 $DATA_DIR/val.source dbart_val_
     --fp16 \
     --bs 32
 ```
-### Multi-GPU Evalulation
+
+
+### Multi-GPU Evaluation
 here is a command to run xsum evaluation on 8 GPUS. It is more than linearly faster than run_eval.py in some cases 
 because it uses SortishSampler to minimize padding. You can also use it on 1 GPU. `data_dir` must have 
 `{type_path}.source` and `{type_path}.target`. Run `python run_distributed_eval.py --help` for all clargs.
@@ -352,3 +354,43 @@ runtime: 13H on V-100 16GB GPU.
 ```bash
 pytest examples/seq2seq/
 ```
+
+### Converting pytorch-lightning checkpoints
+pytorch lightning ``-do_predict`` often fails, after you are done training, the best way to evaluate your model is to convert it.
+
+This should be done for you, with a file called `{save_dir}/best_tfmr`. 
+
+If that file doesn't exist but you have a lightning `.ckpt` file, you can run
+```bash
+python convert_pl_checkpoint_to_hf.py PATH_TO_CKPT  randomly_initialized_hf_model_path save_dir/best_tfmr
+```
+Then either `run_eval` or `run_distributed_eval` with `save_dir/best_tfmr` (see previous sections)
+
+
+## Experimental Features 
+These features are harder to use and not always useful.
+
+###  Dynamic Batch Size for MT
+`finetune.py` has a command line arg `--max_tokens_per_batch` that allows batches to be dynamically sized.
+This feature can only be used:
+- with fairseq installed
+- on 1 GPU
+- without sortish sampler
+- after calling `python save_len_file.py $tok $data_dir`
+
+For example, 
+```bash
+python save_len_file.py Helsinki-NLP/opus-mt-en-ro  wmt_en_ro
+./dynamic_bs_example.sh --max_tokens_per_batch=2000 --output_dir benchmark_dynamic_bs
+```
+splits `wmt_en_ro/train` into 11,197 uneven lengthed batches and can finish 1 epoch in 8 minutes on a v100.
+
+For comparison,
+```bash
+./dynamic_bs_example.sh --sortish_sampler --train_batch_size 48
+```
+uses 12,723 batches of length 48 and takes slightly more time 9.5 minutes.
+
+The feature is still experimental, because:
++ we can make it much more robust if we have memory mapped/preprocessed datasets.
++ The speedup over sortish sampler is not that large at the moment.

examples/seq2seq/__init__.py

@@ -0,0 +1,5 @@
+import os
+import sys
+
+
+sys.path.insert(1, os.path.dirname(os.path.realpath(__file__)))

examples/seq2seq/distillation.py

@@ -10,37 +10,19 @@ import torch
 from torch import nn
 from torch.nn import functional as F
 
-from lightning_base import generic_train
+from finetune import SummarizationModule, TranslationModule
+from finetune import main as ft_main
+from initialization_utils import copy_layers, init_student
 from transformers import AutoModelForSeq2SeqLM, MBartTokenizer, T5Config, T5ForConditionalGeneration
 from transformers.modeling_bart import shift_tokens_right
-
-
-try:
-    from .finetune import SummarizationModule, TranslationModule
-    from .finetune import main as ft_main
-    from .initialization_utils import copy_layers, init_student
-    from .utils import (
-        any_requires_grad,
-        assert_all_frozen,
-        calculate_bleu,
-        freeze_params,
-        label_smoothed_nll_loss,
-        pickle_load,
-        use_task_specific_params,
-    )
-except ImportError:
-    from finetune import SummarizationModule, TranslationModule
-    from finetune import main as ft_main
-    from initialization_utils import copy_layers, init_student
-    from utils import (
-        any_requires_grad,
-        assert_all_frozen,
-        calculate_bleu,
-        freeze_params,
-        label_smoothed_nll_loss,
-        pickle_load,
-        use_task_specific_params,
-    )
+from utils import (
+    any_requires_grad,
+    assert_all_frozen,
+    calculate_bleu,
+    freeze_params,
+    label_smoothed_nll_loss,
+    use_task_specific_params,
+)
 
 
 class BartSummarizationDistiller(SummarizationModule):
@@ -428,30 +410,6 @@ def create_module(args):
     return model
 
 
-def evaluate_checkpoint(ckpt_path: Path, dest_dir=None):
-    # TODO(SS): DELETE? Better to convert_pl_ckpt_to_hf and run_eval.py
-    exp_dir = ckpt_path.parent
-    if dest_dir is None:
-        dest_dir = exp_dir
-    clash = list(dest_dir.glob("test_generations*"))
-    if clash:
-        print(f"SKIPPING to avoid overwriting {clash}")
-    ckpt = torch.load(ckpt_path, map_location="cpu")
-    if "hparams" in ckpt:
-        args = argparse.Namespace(**ckpt["hparams"])
-    else:
-        args = argparse.Namespace(**pickle_load(exp_dir / "hparams.pkl"))
-    args.resume_from_checkpoint = str(ckpt_path)
-    args.do_train = False
-    args.output_dir = str(dest_dir)
-    args.n_gpu = 1
-    args.eval_batch_size = 16
-    Path(args.output_dir).mkdir(exist_ok=True)
-    model = create_module(args)
-    trainer: pl.Trainer = generic_train(model, args, early_stopping_callback=False)
-    trainer.test(model)
-
-
 LAYERS_TO_COPY = {
     # maps  num layers in student -> which teacher layers to copy.
     # 12: bart, 16: pegasus, 6: marian/Helsinki-NLP

examples/seq2seq/dynamic_bs_example.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+export PYTHONPATH="../":"${PYTHONPATH}"
+export WANDB_PROJECT=dmar
+export MAX_LEN=128
+export m=sshleifer/student_marian_en_ro_6_1
+python finetune.py \
+  --learning_rate=3e-4 \
+  --do_train \
+  --fp16 \
+  --data_dir wmt_en_ro \
+  --max_source_length $MAX_LEN --max_target_length $MAX_LEN --val_max_target_length $MAX_LEN --test_max_target_length $MAX_LEN \
+  --freeze_encoder --freeze_embeds \
+  --train_batch_size=48 --eval_batch_size=64 \
+  --tokenizer_name $m --model_name_or_path $m --num_train_epochs=1 \
+  --warmup_steps 500 --logger_name wandb --gpus 1 \
+  --fp16_opt_level=O1 --task translation \
+  "$@"

examples/seq2seq/finetune.py

@@ -12,52 +12,31 @@ import pytorch_lightning as pl
 import torch
 from torch.utils.data import DataLoader
 
+from callbacks import Seq2SeqLoggingCallback, get_checkpoint_callback, get_early_stopping_callback
 from lightning_base import BaseTransformer, add_generic_args, generic_train
 from transformers import MBartTokenizer, T5ForConditionalGeneration
 from transformers.modeling_bart import shift_tokens_right
+from utils import (
+    ROUGE_KEYS,
+    LegacySeq2SeqDataset,
+    Seq2SeqDataset,
+    assert_all_frozen,
+    calculate_bleu,
+    calculate_rouge,
+    flatten_list,
+    freeze_params,
+    get_git_info,
+    label_smoothed_nll_loss,
+    lmap,
+    pickle_save,
+    save_git_info,
+    save_json,
+    use_task_specific_params,
+)
 
 
-try:
-    from .callbacks import Seq2SeqLoggingCallback, get_checkpoint_callback, get_early_stopping_callback
-    from .utils import (
-        ROUGE_KEYS,
-        LegacySeq2SeqDataset,
-        Seq2SeqDataset,
-        assert_all_frozen,
-        calculate_bleu,
-        calculate_rouge,
-        flatten_list,
-        freeze_params,
-        get_git_info,
-        label_smoothed_nll_loss,
-        lmap,
-        pickle_save,
-        save_git_info,
-        save_json,
-        use_task_specific_params,
-    )
-except ImportError:
-    from callbacks import Seq2SeqLoggingCallback, get_checkpoint_callback, get_early_stopping_callback
-    from utils import (
-        ROUGE_KEYS,
-        LegacySeq2SeqDataset,
-        Seq2SeqDataset,
-        assert_all_frozen,
-        calculate_bleu,
-        calculate_rouge,
-        flatten_list,
-        freeze_params,
-        get_git_info,
-        label_smoothed_nll_loss,
-        lmap,
-        pickle_save,
-        save_git_info,
-        save_json,
-        use_task_specific_params,
-    )
-
 logger = logging.getLogger(__name__)
-
+from pytorch_lightning.utilities import rank_zero_only
 
 class SummarizationModule(BaseTransformer):
     mode = "summarization"
@@ -68,6 +47,12 @@ class SummarizationModule(BaseTransformer):
     def __init__(self, hparams, **kwargs):
         if hparams.sortish_sampler and hparams.gpus > 1:
             hparams.replace_sampler_ddp = False
+        elif hparams.max_tokens_per_batch is not None:
+            if hparams.gpus > 1:
+                raise NotImplementedError("Dynamic Batch size does not work for multi-gpu training")
+            if hparams.sortish_sampler:
+                raise ValueError("--sortish_sampler and --max_tokens_per_batch may not be used simultaneously")
+
         super().__init__(hparams, num_labels=None, mode=self.mode, **kwargs)
         use_task_specific_params(self.model, "summarization")
         save_git_info(self.hparams.output_dir)
@@ -96,7 +81,6 @@ class SummarizationModule(BaseTransformer):
         }
         assert self.target_lens["train"] <= self.target_lens["val"], f"target_lens: {self.target_lens}"
         assert self.target_lens["train"] <= self.target_lens["test"], f"target_lens: {self.target_lens}"
-
         if self.hparams.freeze_embeds:
             self.freeze_embeds()
         if self.hparams.freeze_encoder:
@@ -175,6 +159,10 @@ class SummarizationModule(BaseTransformer):
         logs = {name: loss for name, loss in zip(self.loss_names, loss_tensors)}
         # tokens per batch
         logs["tpb"] = batch["input_ids"].ne(self.pad).sum() + batch["labels"].ne(self.pad).sum()
+        logs["bs"] = batch["input_ids"].shape[0]
+        logs["src_pad_tok"] = batch["input_ids"].eq(self.pad).sum()
+        logs["src_pad_frac"] = batch["input_ids"].eq(self.pad).float().mean()
+        # TODO(SS): make a wandb summary metric for this
         return {"loss": loss_tensors[0], "log": logs}
 
     def validation_step(self, batch, batch_idx) -> Dict:
@@ -195,6 +183,20 @@ class SummarizationModule(BaseTransformer):
         losses.update(generative_metrics)
         all_metrics = {f"{prefix}_avg_{k}": x for k, x in losses.items()}
         all_metrics["step_count"] = self.step_count
+
+        def get_date_str(seconds=True) -> str:
+            """Returns 2019-09-25-10:02:07, for example."""
+            if seconds:
+                return time.strftime('%Y-%m-%d-%H:%M:%S')
+            else:
+                return time.strftime('%Y-%m-%d-%H:%M')
+
+        all_metrics['Time'] = get_date_str(seconds=True)
+        #all_metrics['n_obs'] =
+
+
+        all_metrics['rank'] = getattr(self.train_dataloader(), '_rank', -1.)
+
         self.save_metrics(all_metrics, prefix)  # writes to self.metrics_save_path
         preds = flatten_list([x["preds"] for x in outputs])
         return {
@@ -253,20 +255,39 @@ class SummarizationModule(BaseTransformer):
 
     def get_dataloader(self, type_path: str, batch_size: int, shuffle: bool = False) -> DataLoader:
         dataset = self.get_dataset(type_path)
-        sampler = None
-        if self.hparams.sortish_sampler and type_path == "train":
-            sampler = dataset.make_sortish_sampler(batch_size, distributed=self.hparams.gpus > 1)
-            shuffle = False
 
-        dataloader = DataLoader(
-            dataset,
-            batch_size=batch_size,
-            collate_fn=dataset.collate_fn,
-            shuffle=shuffle,
-            num_workers=self.num_workers,
-            sampler=sampler,
-        )
-        return dataloader
+        if self.hparams.sortish_sampler and type_path != "test":
+            sampler = dataset.make_sortish_sampler(batch_size, distributed=self.hparams.gpus > 1)
+            return DataLoader(
+                dataset,
+                batch_size=batch_size,
+                collate_fn=dataset.collate_fn,
+                shuffle=False,
+                num_workers=self.num_workers,
+                sampler=sampler,
+            )
+
+        elif self.hparams.max_tokens_per_batch is not None and type_path != "test":
+            batch_sampler = dataset.make_dynamic_sampler(
+                self.hparams.max_tokens_per_batch, distributed=self.hparams.gpus > 1
+            )
+            return DataLoader(
+                dataset,
+                batch_sampler=batch_sampler,
+                collate_fn=dataset.collate_fn,
+                # shuffle=False,
+                num_workers=self.num_workers,
+                # batch_size=None,
+            )
+        else:
+            return DataLoader(
+                dataset,
+                batch_size=batch_size,
+                collate_fn=dataset.collate_fn,
+                shuffle=shuffle,
+                num_workers=self.num_workers,
+                sampler=None,
+            )
 
     def train_dataloader(self) -> DataLoader:
         dataloader = self.get_dataloader("train", batch_size=self.hparams.train_batch_size, shuffle=True)
@@ -313,6 +334,7 @@ class SummarizationModule(BaseTransformer):
         parser.add_argument("--freeze_encoder", action="store_true")
         parser.add_argument("--freeze_embeds", action="store_true")
         parser.add_argument("--sortish_sampler", action="store_true", default=False)
+        parser.add_argument("--max_tokens_per_batch", type=int, default=None)
         parser.add_argument("--logger_name", type=str, choices=["default", "wandb", "wandb_shared"], default="default")
         parser.add_argument("--n_train", type=int, default=-1, required=False, help="# examples. -1 means use all.")
         parser.add_argument("--n_val", type=int, default=500, required=False, help="# examples. -1 means use all.")

examples/seq2seq/run_distributed_eval.py

@@ -11,35 +11,21 @@ from torch.utils.data import DataLoader
 from tqdm import tqdm
 
 from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+from utils import (
+    Seq2SeqDataset,
+    calculate_bleu,
+    calculate_rouge,
+    lmap,
+    load_json,
+    parse_numeric_n_bool_cl_kwargs,
+    save_json,
+    use_task_specific_params,
+    write_txt_file,
+)
 
 
 logger = getLogger(__name__)
 
-try:
-    from .utils import (
-        Seq2SeqDataset,
-        calculate_bleu,
-        calculate_rouge,
-        lmap,
-        load_json,
-        parse_numeric_n_bool_cl_kwargs,
-        save_json,
-        use_task_specific_params,
-        write_txt_file,
-    )
-except ImportError:
-    from utils import (
-        Seq2SeqDataset,
-        calculate_bleu,
-        calculate_rouge,
-        lmap,
-        load_json,
-        parse_numeric_n_bool_cl_kwargs,
-        save_json,
-        use_task_specific_params,
-        write_txt_file,
-    )
-
 
 def eval_data_dir(
     data_dir,
@@ -176,7 +162,8 @@ def run_generate():
         metrics: Dict = score_fn(preds, labels)
         metrics["n_obs"] = len(preds)
         runtime = time.time() - start_time
-        metrics["seconds_per_sample"] = round(runtime / metrics["n_obs"], 2)
+        metrics["seconds_per_sample"] = round(runtime / metrics["n_obs"], 4)
+        metrics["n_gpus"] = num_replicas
         # TODO(@stas00): add whatever metadata to metrics
         metrics_save_path = save_dir.joinpath(f"{args.type_path}_{metric_name}.json")
         save_json(metrics, metrics_save_path, indent=None)

examples/seq2seq/run_eval.py

@@ -11,14 +11,11 @@ import torch
 from tqdm import tqdm
 
 from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+from utils import calculate_bleu, calculate_rouge, parse_numeric_n_bool_cl_kwargs, use_task_specific_params
 
 
 logger = getLogger(__name__)
 
-try:
-    from .utils import calculate_bleu, calculate_rouge, parse_numeric_n_bool_cl_kwargs, use_task_specific_params
-except ImportError:
-    from utils import calculate_bleu, calculate_rouge, parse_numeric_n_bool_cl_kwargs, use_task_specific_params
 
 DEFAULT_DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
 

examples/seq2seq/run_eval_search.py

@@ -4,18 +4,13 @@ import operator
 import sys
 from collections import OrderedDict
 
-
-try:
-    from .run_eval import datetime_now, run_generate
-except ImportError:
-    from run_eval import datetime_now, run_generate
+from run_eval import datetime_now, run_generate
 
 
 # A table of supported tasks and the list of scores in the order of importance to be sorted by.
 # To add a new task, simply list the score names that `run_eval.run_generate()` returns
 task_score_names = {
     "translation": ["bleu"],
-    "translation_en_to_de": ["bleu"],
     "summarization": ["rouge1", "rouge2", "rougeL"],
 }
 
@@ -66,9 +61,7 @@ def run_search():
     parser.add_argument(
         "--bs", type=int, default=8, required=False, help="initial batch size (may get reduced if it's too big)"
     )
-    parser.add_argument(
-        "--task", type=str, help="used for task_specific_params + metrics", choices=task_score_names.keys()
-    )
+    parser.add_argument("--task", type=str, help="used for task_specific_params + metrics")
     parser.add_argument(
         "--info",
         nargs="?",
@@ -81,8 +74,11 @@ def run_search():
     args_main.extend(["--task", args.task])
     args_normal = [prog] + args_main
 
+    # to support variations like translation_en_to_de"
+    task = "translation" if "translation" in args.task else "summarization"
+
     matrix, col_names = parse_search_arg(args.search)
-    col_names[0:0] = task_score_names[args.task]  # score cols first
+    col_names[0:0] = task_score_names[task]  # score cols first
     col_widths = {col: len(str(col)) for col in col_names}
     results = []
     for r in matrix:
@@ -96,7 +92,7 @@ def run_search():
         scores = run_generate(verbose=False)
         # make sure scores are first in the table
         result = OrderedDict()
-        for score in task_score_names[args.task]:
+        for score in task_score_names[task]:
             result[score] = scores[score]
         result.update(hparams)
         results.append(result)
@@ -107,14 +103,14 @@ def run_search():
             if l > col_widths[k]:
                 col_widths[k] = l
 
-    results_sorted = sorted(results, key=operator.itemgetter(*task_score_names[args.task]), reverse=True)
+    results_sorted = sorted(results, key=operator.itemgetter(*task_score_names[task]), reverse=True)
     print(" | ".join([f"{col:{col_widths[col]}}" for col in col_names]))
     print(" | ".join([f"{'-'*col_widths[col]}" for col in col_names]))
     for row in results_sorted:
         print(" | ".join([f"{row[col]:{col_widths[col]}}" for col in col_names]))
 
     best = results_sorted[0]
-    for score in task_score_names[args.task]:
+    for score in task_score_names[task]:
         del best[score]
     best_args = [f"--{k} {v}" for k, v in best.items()]
     dyn_args = ["--bs", str(args.bs)]

examples/seq2seq/save_len_file.py

@@ -0,0 +1,46 @@
+import fire
+from torch.utils.data import DataLoader
+from tqdm import tqdm
+
+from transformers import AutoTokenizer
+
+
+try:
+    from .utils import Seq2SeqDataset, pickle_save
+except ImportError:
+    from utils import Seq2SeqDataset, pickle_save
+
+
+def save_len_file(
+    tokenizer_name, data_dir, max_source_length=1024, max_target_length=1024, consider_target=False, **kwargs
+):
+    """Save max(src_len, tgt_len) for each example to allow dynamic batching."""
+    tok = AutoTokenizer.from_pretrained(tokenizer_name)
+    train_ds = Seq2SeqDataset(tok, data_dir, max_source_length, max_target_length, type_path="train", **kwargs)
+    pad = tok.pad_token_id
+
+    def get_lens(ds):
+        dl = tqdm(
+            DataLoader(ds, batch_size=512, num_workers=8, shuffle=False, collate_fn=ds.collate_fn),
+            desc=str(ds.len_file),
+        )
+        max_lens = []
+        for batch in dl:
+            src_lens = batch["input_ids"].ne(pad).sum(1).tolist()
+            tgt_lens = batch["labels"].ne(pad).sum(1).tolist()
+            if consider_target:
+                for src, tgt in zip(src_lens, tgt_lens):
+                    max_lens.append(max(src, tgt))
+            else:
+                max_lens.extend(src_lens)
+        return max_lens
+
+    train_lens = get_lens(train_ds)
+    val_ds = Seq2SeqDataset(tok, data_dir, max_source_length, max_target_length, type_path="val", **kwargs)
+    val_lens = get_lens(val_ds)
+    pickle_save(train_lens, train_ds.len_file)
+    pickle_save(val_lens, val_ds.len_file)
+
+
+if __name__ == "__main__":
+    fire.Fire(save_len_file)

examples/seq2seq/test_bash_script.py

@@ -10,13 +10,12 @@ import pytorch_lightning as pl
 import timeout_decorator
 import torch
 
+from distillation import BartSummarizationDistiller, distill_main
+from finetune import SummarizationModule, main
+from test_seq2seq_examples import CUDA_AVAILABLE, MBART_TINY
 from transformers import BartForConditionalGeneration, MarianMTModel
 from transformers.testing_utils import slow
-
-from .distillation import BartSummarizationDistiller, distill_main
-from .finetune import SummarizationModule, main
-from .test_seq2seq_examples import CUDA_AVAILABLE, MBART_TINY
-from .utils import load_json
+from utils import load_json
 
 
 MODEL_NAME = MBART_TINY

examples/seq2seq/test_data/fsmt/build-eval-data.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+
+import io
+import json
+import subprocess
+
+
+pairs = [
+    ["en", "ru"],
+    ["ru", "en"],
+    ["en", "de"],
+    ["de", "en"],
+]
+
+n_objs = 8
+
+
+def get_all_data(pairs, n_objs):
+    text = {}
+    for src, tgt in pairs:
+        pair = f"{src}-{tgt}"
+        cmd = f"sacrebleu -t wmt19 -l {pair} --echo src".split()
+        src_lines = subprocess.run(cmd, stdout=subprocess.PIPE).stdout.decode("utf-8").splitlines()
+        cmd = f"sacrebleu -t wmt19 -l {pair} --echo ref".split()
+        tgt_lines = subprocess.run(cmd, stdout=subprocess.PIPE).stdout.decode("utf-8").splitlines()
+        text[pair] = {"src": src_lines[:n_objs], "tgt": tgt_lines[:n_objs]}
+    return text
+
+
+text = get_all_data(pairs, n_objs)
+filename = "./fsmt_val_data.json"
+with io.open(filename, "w", encoding="utf-8") as f:
+    bleu_data = json.dump(text, f, indent=2, ensure_ascii=False)

examples/seq2seq/test_data/fsmt/fsmt_val_data.json

@@ -0,0 +1,90 @@
+{
+  "en-ru": {
+    "src": [
+      "Welsh AMs worried about 'looking like muppets'",
+      "There is consternation among some AMs at a suggestion their title should change to MWPs (Member of the Welsh Parliament).",
+      "It has arisen because of plans to change the name of the assembly to the Welsh Parliament.",
+      "AMs across the political spectrum are worried it could invite ridicule.",
+      "One Labour AM said his group was concerned \"it rhymes with Twp and Pwp.\"",
+      "For readers outside of Wales: In Welsh twp means daft and pwp means poo.",
+      "A Plaid AM said the group as a whole was \"not happy\" and has suggested alternatives.",
+      "A Welsh Conservative said his group was \"open minded\" about the name change, but noted it was a short verbal hop from MWP to Muppet."
+    ],
+    "tgt": [
+      "Члены Национальной ассамблеи Уэльса обеспокоены, что \"выглядят как куклы\"",
+      "Некоторые члены Национальной ассамблеи Уэльса в ужасе от предложения о том, что их наименование должно измениться на MPW (члены Парламента Уэльса).",
+      "Этот вопрос был поднят в связи с планами по переименованию ассамблеи в Парламент Уэльса.",
+      "Члены Национальной ассамблеи Уэльса всего политического спектра обеспокоены, что это может породить насмешки.",
+      "Один из лейбористских членов Национальной ассамблеи Уэльса сказал, что его партия обеспокоена тем, что \"это рифмуется с Twp и Pwp\".",
+      "Для читателей за предлами Уэльса: по-валлийски twp означает \"глупый\", а pwp означает \"какашка\".",
+      "Член Национальной ассамблеи от Плайд сказал, что эта партия в целом \"не счастлива\" и предложил альтернативы.",
+      "Представитель Консервативной партии Уэльса сказал, что его партия \"открыта\" к переименованию, но отметил, что между WMP и Muppet небольшая разница в произношении."
+    ]
+  },
+  "ru-en": {
+    "src": [
+      "Названо число готовящихся к отправке в Донбасс новобранцев из Украины",
+      "Официальный представитель Народной милиции самопровозглашенной Луганской Народной Республики (ЛНР) Андрей Марочко заявил, что зимой 2018-2019 года Украина направит в Донбасс не менее 3 тыс. новобранцев.",
+      "По его словам, таким образом Киев планирует \"хоть как-то доукомплектовать подразделения\".",
+      "\"Нежелание граждан Украины проходить службу в рядах ВС Украины, массовые увольнения привели к низкой укомплектованности подразделений\", - рассказал Марочко, которого цитирует \"РИА Новости\".",
+      "Он также не исключил, что реальные цифры призванных в армию украинцев могут быть увеличены в случае необходимости.",
+      "В 2014-2017 годах Киев начал так называемую антитеррористическую операцию (АТО), которую позже сменили на операцию объединенных сил (ООС).",
+      "Предполагалось, что эта мера приведет к усилению роли украинских силовиков в урегулировании ситуации.",
+      "В конце августа 2018 года ситуация в Донбассе обострилась из-за убийства главы ДНР Александра Захарченко."
+    ],
+    "tgt": [
+      "The number of new Ukrainian recruits ready to go to Donbass has become public",
+      "Official representative of the peoples’ militia of the self-proclaimed Lugansk People’s Republic Andrey Marochko claimed that Ukrainian will send at least 3 thousand new recruits to Donbass in winter 2018-2019.",
+      "This is how Kyiv tries “at least somehow to staff the units,” he said.",
+      "“The unwillingness of Ukrainian citizens to serve in the Ukraine’s military forces, mass resignments lead to low understaffing,” said Marochko cited by RIA Novosti.",
+      "Also, he doesn’t exclude that the real numbers of conscripts in the Ukrainian army can be raised is necessary.",
+      "In 2014-2017, Kyiv started so-called antiterrorist operation, that ws later changed to the united forces operation.",
+      "This measure was supposed to strengthen the role of the Ukrainian military in settling the situation.",
+      "In the late August 2018, the situation in Donbass escalated as the DNR head Aleksandr Zakharchenko was killed."
+    ]
+  },
+  "en-de": {
+    "src": [
+      "Welsh AMs worried about 'looking like muppets'",
+      "There is consternation among some AMs at a suggestion their title should change to MWPs (Member of the Welsh Parliament).",
+      "It has arisen because of plans to change the name of the assembly to the Welsh Parliament.",
+      "AMs across the political spectrum are worried it could invite ridicule.",
+      "One Labour AM said his group was concerned \"it rhymes with Twp and Pwp.\"",
+      "For readers outside of Wales: In Welsh twp means daft and pwp means poo.",
+      "A Plaid AM said the group as a whole was \"not happy\" and has suggested alternatives.",
+      "A Welsh Conservative said his group was \"open minded\" about the name change, but noted it was a short verbal hop from MWP to Muppet."
+    ],
+    "tgt": [
+      "Walisische Ageordnete sorgen sich \"wie Dödel auszusehen\"",
+      "Es herrscht Bestürzung unter einigen Mitgliedern der Versammlung über einen Vorschlag, der ihren Titel zu MWPs (Mitglied der walisischen Parlament) ändern soll.",
+      "Der Grund dafür waren Pläne, den Namen der Nationalversammlung in Walisisches Parlament zu ändern.",
+      "Mitglieder aller Parteien der Nationalversammlung haben Bedenken, dass sie sich dadurch Spott aussetzen könnten.",
+      "Ein Labour-Abgeordneter sagte, dass seine Gruppe \"sich mit Twp und Pwp reimt\".",
+      "Hinweis für den Leser: „twp“ im Walisischen bedeutet „bescheuert“ und „pwp“ bedeutet „Kacke“.",
+      "Ein Versammlungsmitglied von Plaid Cymru sagte, die Gruppe als Ganzes sei \"nicht glücklich\" und hat Alternativen vorgeschlagen.",
+      "Ein walisischer Konservativer sagte, seine Gruppe wäre „offen“ für eine Namensänderung, wies aber darauf hin, dass es von „MWP“ (Mitglied des Walisischen Parlaments) nur ein kurzer verbaler Sprung zu „Muppet“ ist."
+    ]
+  },
+  "de-en": {
+    "src": [
+      "Schöne Münchnerin 2018: Schöne Münchnerin 2018 in Hvar: Neun Dates",
+      "Von az, aktualisiert am 04.05.2018 um 11:11",
+      "Ja, sie will...",
+      "\"Schöne Münchnerin\" 2018 werden!",
+      "Am Nachmittag wartet erneut eine Überraschung auf unsere Kandidatinnen: sie werden das romantische Candlelight-Shooting vor der MY SOLARIS nicht alleine bestreiten, sondern an der Seite von Male-Model Fabian!",
+      "Hvar - Flirten, kokettieren, verführen - keine einfachen Aufgaben für unsere Mädchen.",
+      "Insbesondere dann, wenn in Deutschland ein Freund wartet.",
+      "Dennoch liefern die neun \"Schöne Münchnerin\"-Kandidatinnen beim Shooting mit People-Fotograf Tuan ab und trotzen Wind, Gischt und Regen wie echte Profis."
+    ],
+    "tgt": [
+      "The Beauty of Munich 2018: the Beauty of Munich 2018 in Hvar: Nine dates",
+      "From A-Z, updated on 04/05/2018 at 11:11",
+      "Yes, she wants to...",
+      "to become \"The Beauty of Munich\" in 2018!",
+      "In the afternoon there is another surprise waiting for our contestants: they will be competing for the romantic candlelight photo shoot at MY SOLARIS not alone, but together with a male-model Fabian!",
+      "Hvar with its flirting, coquetting, and seduction is not an easy task for our girls.",
+      "Especially when there is a boyfriend waiting in Germany.",
+      "Despite dealing with wind, sprays and rain, the nine contestants of \"The Beauty of Munich\" behaved like real professionals at the photo shoot with People-photographer Tuan."
+    ]
+  }
+}

examples/seq2seq/test_data/wmt_en_ro/train.source

@@ -1,8 +1,11 @@
+Corrections to votes and voting intentions: see Minutes Assignment conferred on a Member: see Minutes Membership of committees and delegations: see Minutes Decisions concerning certain documents: see Minutes Forwarding of texts adopted during the sitting: see Minutes Dates for next sittings: see Minutes
 Membership of Parliament: see Minutes Approval of Minutes of previous sitting: see Minutes Membership of Parliament: see Minutes Verification of credentials: see Minutes Documents received: see Minutes Written statements and oral questions (tabling): see Minutes Petitions: see Minutes Texts of agreements forwarded by the Council: see Minutes Action taken on Parliament's resolutions: see Minutes Agenda for next sitting: see Minutes Closure of sitting (The sitting was closed at 7.45 p.m.)
 Election of Vice-Presidents of the European Parliament (deadline for submitting nominations): see Minutes (The sitting was suspended at 12.40 p.m. and resumed at 3.00 p.m.) Election of Quaestors of the European Parliament (deadline for submitting nominations): see Minutes (The sitting was suspended at 3.25 p.m. and resumed at 6.00 p.m.) Agenda for next sitting: see Minutes Closure of sitting (The sitting was closed at 6.15 p.m.) Opening of the sitting (The sitting was opened at 9.35 a.m.) Documents received: see Minutes Approval of Minutes of previous sitting: see Minutes Membership of Parliament: see Minutes
 Membership of committees (deadline for tabling amendments): see Minutes (The sitting was suspended at 7 p.m. and resumed at 9 p.m.) Agenda for next sitting: see Minutes Closure of sitting (The sitting was suspended at 23.25 p.m.) Documents received: see Minutes Communication of Council common positions: see Minutes (The sitting was suspended at 11.35 a.m. and resumed for voting time at noon) Approval of Minutes of previous sitting: see Minutes Committee of Inquiry into the crisis of the Equitable Life Assurance Society (extension of mandate): see Minutes
 Announcement by the President: see Minutes 1. Membership of committees (vote) 2. Amendment of the ACP-EC Partnership Agreement (vote) 4. Certification of train drivers operating locomotives and trains on the railway system in the Community (vote) 6. Law applicable to non-contractual obligations ("ROME II") (vote) 8. Seventh and eighth annual reports on arms exports (vote) Corrections to votes and voting intentions: see Minutes Membership of committees and delegations: see Minutes Request for waiver of parliamentary immunity: see Minutes Decisions concerning certain documents: see Minutes
+Written statements for entry
 Written statements for entry in the register (Rule 116): see Minutes Forwarding of texts adopted during the sitting: see Minutes Dates for next sittings: see Minutes Adjournment of the session I declare the session of the European Parliament adjourned. (The sitting was closed at 1 p.m.) Approval of Minutes of previous sitting: see Minutes Membership of Parliament: see Minutes Request for the defence of parliamentary immunity: see Minutes Appointments to committees (proposal by the Conference of Presidents): see Minutes Documents received: see Minutes Texts of agreements forwarded by the Council: see Minutes
 Action taken on Parliament's resolutions: see Minutes Oral questions and written statements (tabling): see Minutes Written statements (Rule 116): see Minutes Agenda: see Minutes 1. Appointments to parliamentary committees (vote): see Minutes Voting time Agenda for next sitting: see Minutes Closure of sitting (The sitting was closed at 12 midnight) Opening of the sitting (The sitting was opened at 09.05) Documents received: see Minutes Approval of Minutes of previous sitting: see Minutes 1. Protection of passengers against displaced luggage (vote) 2.
 Approval of motor vehicles with regard to the forward field of vision of the driver (vote) 3. EC-Korea Agreement on scientific and technological cooperation (vote) 4. Mainstreaming sustainability in development cooperation policies (vote) 5. Draft Amending Budget No 1/2007 (vote) 7. EC-Gabon Fisheries Partnership (vote) 10. Limitation periods in cross-border disputes involving personal injuries and fatal accidents (vote) 12. Strategy for a strengthened partnership with the Pacific Islands (vote) 13. The European private company statute (vote) That concludes the vote.
-Corrections to votes and voting intentions: see Minutes Assignment conferred on a Member: see Minutes Membership of committees and delegations: see Minutes Decisions concerning certain documents: see Minutes Forwarding of texts adopted during the sitting: see Minutes Dates for next sittings: see Minutes
+Corrections to votes and voting intentions: see Minutes Assignment conferred on a Member: see Minutes Membership of committees and delegations: see Minutes Decisions concerning certain documents: see Minutes Forwarding of texts adopted during the sitting: see Minutes Dates for next sittings: see Minutes
+Written statements for entry

examples/seq2seq/test_data/wmt_en_ro/train.target

@@ -1,8 +1,11 @@
+Corectările voturilor şi intenţiile de vot: a se vedea procesul-verbal Misiune încredinţată unui deputat: consultaţi procesul-verbal Componenţa comisiilor şi a delegaţiilor: a se vedea procesul-verbal Decizii privind anumite documente: a se vedea procesul-verbal Transmiterea textelor adoptate în cursul prezentei şedinţe: a se vedea procesul-verbal Calendarul următoarelor şedinţe: a se vedea procesul-verbal
 Componenţa Parlamentului: a se vedea procesul-verbal Aprobarea procesului-verbal al şedinţei precedente: a se vedea procesul-verbal Componenţa Parlamentului: a se vedea procesul-verbal Verificarea prerogativelor: a se vedea procesul-verbal Depunere de documente: a se vedea procesul-verbal Declaraţii scrise şi întrebări orale (depunere): consultaţi procesul-verbal Petiţii: a se vedea procesul-verbal Transmiterea de către Consiliu a textelor acordurilor: a se vedea procesul-verbal Cursul dat rezoluţiilor Parlamentului: a se vedea procesul-verbal Ordinea de zi a următoarei şedinţe: a se vedea procesul-verbal Ridicarea şedinţei (Se levanta la sesión a las 19.45 horas)
 Alegerea vicepreşedinţilor Parlamentului European (termenul de depunere a candidaturilor): consultaţi procesul-verbal (Die Sitzung wird um 12.40 Uhr unterbrochen und um 15.00 Uhr wiederaufgenommen). Alegerea chestorilor Parlamentului European (termenul de depunere a candidaturilor): consultaţi procesul-verbal (Die Sitzung wird um 15.25 Uhr unterbrochen und um 18.00 Uhr wiederaufgenommen). Ordinea de zi a următoarei şedinţe: a se vedea procesul-verbal Ridicarea şedinţei (Die Sitzung wird um 18.15 Uhr geschlossen.) Deschiderea şedinţei (Die Sitzung wird um 9.35 Uhr eröffnet.) Depunerea documentelor: a se vedea procesul-verbal Aprobarea procesului-verbal al şedinţei precedente: a se vedea procesul-verbal Componenţa Parlamentului: a se vedea procesul-verbal
 Componenţa comisiilor (termenul de depunere a amendamentelor): consultaţi procesul-verbal (La seduta, sospesa alle 19.00, è ripresa alle 21.00) Ordinea de zi a următoarei şedinţe: a se vedea procesul-verbal Ridicarea şedinţei (Die Sitzung wird um 23.25 Uhr geschlossen.) Depunerea documentelor: a se vedea procesul-verbal Comunicarea poziţiilor comune ale Parlamentului: a se vedea procesul-verbal (La séance, suspendue à 11h35 dans l'attente de l'Heure des votes, est reprise à midi) Aprobarea procesului-verbal al şedinţei precedente: a se vedea procesul-verbal Comisia de anchetă privind criza societăţii de asigurări "Equitable Life” (prelungirea mandatului): consultaţi procesul-verbal
 Comunicarea Preşedintelui: consultaţi procesul-verbal 1. Componenţa comisiilor (vot) 2. Modificarea Acordului de parteneriat ACP-CE ("Acordul de la Cotonou”) (vot) 4. Certificarea mecanicilor de locomotivă care conduc locomotive şi trenuri în sistemul feroviar comunitar (vot) 6. Legea aplicabilă obligaţiilor necontractuale ("Roma II”) (vot) 8. Al şaptelea şi al optulea raport anual privind exportul de armament (vot) Corectările voturilor şi intenţiile de vot: a se vedea procesul-verbal Componenţa comisiilor şi a delegaţiilor: a se vedea procesul-verbal Cerere de ridicare a imunităţii parlamentare: consultaţi procesul-verbal Decizii privind anumite documente: a se vedea procesul-verbal
+Declaraţii scrise înscrise
 Declaraţii scrise înscrise în registru (articolul 116 din Regulamentul de procedură): a se vedea procesul-verbal Transmiterea textelor adoptate în cursul prezentei şedinţe: a se vedea procesul-verbal Calendarul următoarelor şedinţe: a se vedea procesul-verbal Întreruperea sesiunii Dichiaro interrotta la sessione del Parlamento europeo. (La seduta è tolta alle 13.00) Aprobarea procesului-verbal al şedinţei precedente: a se vedea procesul-verbal Componenţa Parlamentului: a se vedea procesul-verbal Cerere de apărare a imunităţii parlamentare: consultaţi procesul-verbal Numiri în comisii (propunerea Conferinţei preşedinţilor): consultaţi procesul-verbal Depunerea documentelor: a se vedea procesul-verbal Transmiterea de către Consiliu a textelor acordurilor: a se vedea procesul-verbal
 Continuări ale rezoluţiilor Parlamentului: consultaţi procesul-verbal Declaraţii scrise şi întrebări orale (depunere): consultaţi procesul-verbal Declaraţii scrise (articolul 116 din Regulamentul de procedură) Ordinea de zi: a se vedea procesul-verbal 1. Numiri în comisiile parlamentare (vot): consultaţi procesul-verbal Timpul afectat votului Ordinea de zi a următoarei şedinţe: a se vedea procesul-verbal Ridicarea şedinţei (La seduta è tolta alle 24.00) Deschiderea şedinţei (The sitting was opened at 09.05) Depunerea documentelor: a se vedea procesul-verbal Aprobarea procesului-verbal al şedinţei precedente: a se vedea procesul-verbal 1. Protecţia pasagerilor împotriva deplasării bagajelor (vot) 2.
 Omologarea vehiculelor cu motor cu privire la câmpul de vizibilitate înainte al conducătorului auto (vot) 3. Acordul CE-Coreea de cooperare ştiinţifică şi tehnologică (vot) 4. Integrarea durabilităţii în politicile de cooperare pentru dezvoltare (vot) 5. Proiect de buget rectificativ nr.1/2007 (vot) 7. Acordul de parteneriat în domeniul pescuitului între Comunitatea Europeană şi Republica Gaboneză (vot) 10. Termenele de prescripţie aplicabile în cadrul litigiilor transfrontaliere cu privire la vătămările corporale şi accidentele mortale (vot) 12. Relaţiile UE cu insulele din Pacific: Strategie pentru un parteneriat consolidat (vot) 13. Statutul societăţii private europene (vot) Damit ist die Abstimmungsstunde beendet.
-Corectările voturilor şi intenţiile de vot: a se vedea procesul-verbal Misiune încredinţată unui deputat: consultaţi procesul-verbal Componenţa comisiilor şi a delegaţiilor: a se vedea procesul-verbal Decizii privind anumite documente: a se vedea procesul-verbal Transmiterea textelor adoptate în cursul prezentei şedinţe: a se vedea procesul-verbal Calendarul următoarelor şedinţe: a se vedea procesul-verbal
+Corectările voturilor şi intenţiile de vot: a se vedea procesul-verbal Misiune încredinţată unui deputat: consultaţi procesul-verbal Componenţa comisiilor şi a delegaţiilor: a se vedea procesul-verbal Decizii privind anumite documente: a se vedea procesul-verbal Transmiterea textelor adoptate în cursul prezentei şedinţe: a se vedea procesul-verbal Calendarul următoarelor şedinţe: a se vedea procesul-verbal
+Declaraţii scrise înscrise

examples/seq2seq/test_datasets.py

@@ -0,0 +1,188 @@
+import os
+import tempfile
+from pathlib import Path
+
+import numpy as np
+import pytest
+from torch.utils.data import DataLoader
+
+from transformers import AutoTokenizer
+from transformers.modeling_bart import shift_tokens_right
+from transformers.testing_utils import slow
+
+from .pack_dataset import pack_data_dir
+from .save_len_file import save_len_file
+from .test_seq2seq_examples import ARTICLES, BART_TINY, MARIAN_TINY, MBART_TINY, SUMMARIES, T5_TINY, make_test_data_dir
+from .utils import FAIRSEQ_AVAILABLE, DistributedSortishSampler, LegacySeq2SeqDataset, Seq2SeqDataset
+
+
+BERT_BASE_CASED = "bert-base-cased"
+PEGASUS_XSUM = "google/pegasus-xsum"
+
+
+@slow
+@pytest.mark.parametrize(
+    "tok_name",
+    [
+        MBART_TINY,
+        MARIAN_TINY,
+        T5_TINY,
+        BART_TINY,
+        PEGASUS_XSUM,
+    ],
+)
+def test_seq2seq_dataset_truncation(tok_name):
+    tokenizer = AutoTokenizer.from_pretrained(tok_name)
+    tmp_dir = make_test_data_dir()
+    max_len_source = max(len(tokenizer.encode(a)) for a in ARTICLES)
+    max_len_target = max(len(tokenizer.encode(a)) for a in SUMMARIES)
+    max_src_len = 4
+    max_tgt_len = 8
+    assert max_len_target > max_src_len  # Will be truncated
+    assert max_len_source > max_src_len  # Will be truncated
+    src_lang, tgt_lang = "ro_RO", "de_DE"  # ignored for all but mbart, but never causes error.
+    train_dataset = Seq2SeqDataset(
+        tokenizer,
+        data_dir=tmp_dir,
+        type_path="train",
+        max_source_length=max_src_len,
+        max_target_length=max_tgt_len,  # ignored
+        src_lang=src_lang,
+        tgt_lang=tgt_lang,
+    )
+    dataloader = DataLoader(train_dataset, batch_size=2, collate_fn=train_dataset.collate_fn)
+    for batch in dataloader:
+        assert isinstance(batch, dict)
+        assert batch["attention_mask"].shape == batch["input_ids"].shape
+        # show that articles were trimmed.
+        assert batch["input_ids"].shape[1] == max_src_len
+        # show that targets are the same len
+        assert batch["labels"].shape[1] == max_tgt_len
+        if tok_name != MBART_TINY:
+            continue
+        # check language codes in correct place
+        batch["decoder_input_ids"] = shift_tokens_right(batch["labels"], tokenizer.pad_token_id)
+        assert batch["decoder_input_ids"][0, 0].item() == tokenizer.lang_code_to_id[tgt_lang]
+        assert batch["decoder_input_ids"][0, -1].item() == tokenizer.eos_token_id
+        assert batch["input_ids"][0, -2].item() == tokenizer.eos_token_id
+        assert batch["input_ids"][0, -1].item() == tokenizer.lang_code_to_id[src_lang]
+
+        break  # No need to test every batch
+
+
+@pytest.mark.parametrize("tok", [BART_TINY, BERT_BASE_CASED])
+def test_legacy_dataset_truncation(tok):
+    tokenizer = AutoTokenizer.from_pretrained(tok)
+    tmp_dir = make_test_data_dir()
+    max_len_source = max(len(tokenizer.encode(a)) for a in ARTICLES)
+    max_len_target = max(len(tokenizer.encode(a)) for a in SUMMARIES)
+    trunc_target = 4
+    train_dataset = LegacySeq2SeqDataset(
+        tokenizer,
+        data_dir=tmp_dir,
+        type_path="train",
+        max_source_length=20,
+        max_target_length=trunc_target,
+    )
+    dataloader = DataLoader(train_dataset, batch_size=2, collate_fn=train_dataset.collate_fn)
+    for batch in dataloader:
+        assert batch["attention_mask"].shape == batch["input_ids"].shape
+        # show that articles were trimmed.
+        assert batch["input_ids"].shape[1] == max_len_source
+        assert 20 >= batch["input_ids"].shape[1]  # trimmed significantly
+        # show that targets were truncated
+        assert batch["labels"].shape[1] == trunc_target  # Truncated
+        assert max_len_target > trunc_target  # Truncated
+        break  # No need to test every batch
+
+
+def test_pack_dataset():
+    tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")
+
+    tmp_dir = Path(make_test_data_dir())
+    orig_examples = tmp_dir.joinpath("train.source").open().readlines()
+    save_dir = Path(tempfile.mkdtemp(prefix="packed_"))
+    pack_data_dir(tokenizer, tmp_dir, 128, save_dir)
+    orig_paths = {x.name for x in tmp_dir.iterdir()}
+    new_paths = {x.name for x in save_dir.iterdir()}
+    packed_examples = save_dir.joinpath("train.source").open().readlines()
+    # orig: [' Sam ate lunch today.\n', 'Sams lunch ingredients.']
+    # desired_packed: [' Sam ate lunch today.\n Sams lunch ingredients.']
+    assert len(packed_examples) < len(orig_examples)
+    assert len(packed_examples) == 1
+    assert len(packed_examples[0]) == sum(len(x) for x in orig_examples)
+    assert orig_paths == new_paths
+
+
+@pytest.mark.skipif(not FAIRSEQ_AVAILABLE, reason="This test requires fairseq")
+def test_dynamic_batch_size():
+    if not FAIRSEQ_AVAILABLE:
+        return
+    ds, max_tokens, tokenizer = _get_dataset(max_len=64)
+    required_batch_size_multiple = 64
+    batch_sampler = ds.make_dynamic_sampler(max_tokens, required_batch_size_multiple=required_batch_size_multiple)
+    batch_sizes = [len(x) for x in batch_sampler]
+    assert len(set(batch_sizes)) > 1  # it's not dynamic batch size if every batch is the same length
+    assert sum(batch_sizes) == len(ds)  # no dropped or added examples
+    data_loader = DataLoader(ds, batch_sampler=batch_sampler, collate_fn=ds.collate_fn, num_workers=2)
+    failures = []
+    num_src_per_batch = []
+    for batch in data_loader:
+        src_shape = batch["input_ids"].shape
+        bs = src_shape[0]
+        assert bs % required_batch_size_multiple == 0 or bs < required_batch_size_multiple
+        num_src_tokens = np.product(batch["input_ids"].shape)
+        num_src_per_batch.append(num_src_tokens)
+        if num_src_tokens > (max_tokens * 1.1):
+            failures.append(num_src_tokens)
+    assert num_src_per_batch[0] == max(num_src_per_batch)
+    if failures:
+        raise AssertionError(f"too many tokens in {len(failures)} batches")
+
+
+def test_sortish_sampler_reduces_padding():
+    ds, _, tokenizer = _get_dataset(max_len=512)
+    bs = 2
+    sortish_sampler = ds.make_sortish_sampler(bs, shuffle=False)
+
+    naive_dl = DataLoader(ds, batch_size=bs, collate_fn=ds.collate_fn, num_workers=2)
+    sortish_dl = DataLoader(ds, batch_size=bs, collate_fn=ds.collate_fn, num_workers=2, sampler=sortish_sampler)
+
+    pad = tokenizer.pad_token_id
+
+    def count_pad_tokens(data_loader, k="input_ids"):
+        return [batch[k].eq(pad).sum().item() for batch in data_loader]
+
+    assert sum(count_pad_tokens(sortish_dl, k="labels")) < sum(count_pad_tokens(naive_dl, k="labels"))
+    assert sum(count_pad_tokens(sortish_dl)) < sum(count_pad_tokens(naive_dl))
+    assert len(sortish_dl) == len(naive_dl)
+
+
+def _get_dataset(n_obs=1000, max_len=128):
+    if os.getenv("USE_REAL_DATA", False):
+        data_dir = "examples/seq2seq/wmt_en_ro"
+        max_tokens = max_len * 2 * 64
+        if not Path(data_dir).joinpath("train.len").exists():
+            save_len_file(MARIAN_TINY, data_dir)
+    else:
+        data_dir = "examples/seq2seq/test_data/wmt_en_ro"
+        max_tokens = max_len * 4
+        save_len_file(MARIAN_TINY, data_dir)
+
+    tokenizer = AutoTokenizer.from_pretrained(MARIAN_TINY)
+    ds = Seq2SeqDataset(
+        tokenizer,
+        data_dir=data_dir,
+        type_path="train",
+        max_source_length=max_len,
+        max_target_length=max_len,
+        n_obs=n_obs,
+    )
+    return ds, max_tokens, tokenizer
+
+
+def test_distributed_sortish_sampler_splits_indices_between_procs():
+    ds, max_tokens, tokenizer = _get_dataset()
+    ids1 = set(DistributedSortishSampler(ds, 256, num_replicas=2, rank=0, add_extra_examples=False))
+    ids2 = set(DistributedSortishSampler(ds, 256, num_replicas=2, rank=1, add_extra_examples=False))
+    assert ids1.intersection(ids2) == set()

examples/seq2seq/test_fsmt_bleu_score.py

@@ -0,0 +1,77 @@
+# coding=utf-8
+# Copyright 2020 Huggingface
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import io
+import unittest
+
+
+try:
+    from .utils import calculate_bleu
+except ImportError:
+    from utils import calculate_bleu
+
+import json
+
+from parameterized import parameterized
+from transformers import FSMTForConditionalGeneration, FSMTTokenizer
+from transformers.testing_utils import get_tests_dir, require_torch, slow, torch_device
+
+
+filename = get_tests_dir() + "/test_data/fsmt/fsmt_val_data.json"
+with io.open(filename, "r", encoding="utf-8") as f:
+    bleu_data = json.load(f)
+
+
+@require_torch
+class ModelEvalTester(unittest.TestCase):
+    def get_tokenizer(self, mname):
+        return FSMTTokenizer.from_pretrained(mname)
+
+    def get_model(self, mname):
+        model = FSMTForConditionalGeneration.from_pretrained(mname).to(torch_device)
+        if torch_device == "cuda":
+            model.half()
+        return model
+
+    @parameterized.expand(
+        [
+            ["en-ru", 26.0],
+            ["ru-en", 22.0],
+            ["en-de", 22.0],
+            ["de-en", 29.0],
+        ]
+    )
+    @slow
+    def test_bleu_scores(self, pair, min_bleu_score):
+        # note: this test is not testing the best performance since it only evals a small batch
+        # but it should be enough to detect a regression in the output quality
+        mname = f"facebook/wmt19-{pair}"
+        tokenizer = self.get_tokenizer(mname)
+        model = self.get_model(mname)
+
+        src_sentences = bleu_data[pair]["src"]
+        tgt_sentences = bleu_data[pair]["tgt"]
+
+        batch = tokenizer(src_sentences, return_tensors="pt", truncation=True, padding="longest").to(torch_device)
+        outputs = model.generate(
+            input_ids=batch.input_ids,
+            num_beams=8,
+        )
+        decoded_sentences = tokenizer.batch_decode(
+            outputs, skip_special_tokens=True, clean_up_tokenization_spaces=False
+        )
+        scores = calculate_bleu(decoded_sentences, tgt_sentences)
+        print(scores)
+        self.assertGreaterEqual(scores["bleu"], min_bleu_score)

examples/seq2seq/test_seq2seq_examples.py

@@ -10,21 +10,17 @@ from unittest.mock import patch
 import pytest
 import pytorch_lightning as pl
 import torch
-from torch.utils.data import DataLoader
 
 import lightning_base
-from transformers import AutoConfig, AutoModelForSeq2SeqLM, AutoTokenizer
+from convert_pl_checkpoint_to_hf import convert_pl_to_hf
+from distillation import distill_main
+from finetune import SummarizationModule, main
+from run_eval import generate_summaries_or_translations, run_generate
+from run_eval_search import run_search
+from transformers import AutoConfig, AutoModelForSeq2SeqLM
 from transformers.hf_api import HfApi
-from transformers.modeling_bart import shift_tokens_right
 from transformers.testing_utils import CaptureStderr, CaptureStdout, require_multigpu, require_torch_and_cuda, slow
-
-from .convert_pl_checkpoint_to_hf import convert_pl_to_hf
-from .distillation import distill_main, evaluate_checkpoint
-from .finetune import SummarizationModule, main
-from .pack_dataset import pack_data_dir
-from .run_eval import generate_summaries_or_translations, run_generate
-from .run_eval_search import run_search
-from .utils import LegacySeq2SeqDataset, Seq2SeqDataset, label_smoothed_nll_loss, lmap, load_json
+from utils import label_smoothed_nll_loss, lmap, load_json
 
 
 logging.basicConfig(level=logging.DEBUG)
@@ -32,6 +28,7 @@ logging.basicConfig(level=logging.DEBUG)
 logger = logging.getLogger()
 CUDA_AVAILABLE = torch.cuda.is_available()
 CHEAP_ARGS = {
+    "max_tokens_per_batch": None,
     "supervise_forward": True,
     "normalize_hidden": True,
     "label_smoothing": 0.2,
@@ -106,6 +103,8 @@ T5_TINY = "patrickvonplaten/t5-tiny-random"
 BART_TINY = "sshleifer/bart-tiny-random"
 MBART_TINY = "sshleifer/tiny-mbart"
 MARIAN_TINY = "sshleifer/tiny-marian-en-de"
+
+
 stream_handler = logging.StreamHandler(sys.stdout)
 logger.addHandler(stream_handler)
 logging.disable(logging.CRITICAL)  # remove noisy download output from tracebacks
@@ -178,7 +177,6 @@ class TestSummarizationDistiller(unittest.TestCase):
         generate_summaries_or_translations(examples, out_path, str(model.output_dir / "best_tfmr"))
         self.assertTrue(Path(out_path).exists())
 
-        evaluate_checkpoint(ckpts[0], dest_dir=Path(tempfile.mkdtemp()))
         out_path_new = tempfile.mkdtemp()
         convert_pl_to_hf(ckpts[0], transformer_ckpts[0].parent, out_path_new)
         assert os.path.exists(os.path.join(out_path_new, "pytorch_model.bin"))
@@ -227,8 +225,6 @@ class TestSummarizationDistiller(unittest.TestCase):
         assert len(all_files) > 2
         self.assertEqual(len(transformer_ckpts), 2)
 
-        evaluate_checkpoint(ckpts[0], dest_dir=Path(tempfile.mkdtemp()))
-
     @unittest.skip("T5 distillation is broken at the moment")
     def test_distill_t5(self):
         updates = dict(
@@ -284,8 +280,7 @@ class TestSummarizationDistiller(unittest.TestCase):
         return model
 
 
-@pytest.mark.parametrize("model", [pytest.param(T5_TINY), pytest.param(BART_TINY), pytest.param(MBART_TINY)])
-def test_run_eval(model):
+def run_eval_tester(model):
     input_file_name = Path(tempfile.mkdtemp()) / "utest_input.source"
     output_file_name = input_file_name.parent / "utest_output.txt"
     assert not output_file_name.exists()
@@ -293,28 +288,39 @@ def test_run_eval(model):
     _dump_articles(input_file_name, articles)
     score_path = str(Path(tempfile.mkdtemp()) / "scores.json")
     task = "translation_en_to_de" if model == T5_TINY else "summarization"
-    testargs = [
-        "run_eval.py",
-        model,
-        str(input_file_name),
-        str(output_file_name),
-        "--score_path",
-        score_path,
-        "--task",
-        task,
-        "--num_beams",
-        "2",
-        "--length_penalty",
-        "2.0",
-    ]
+    testargs = f"""
+        run_eval_search.py
+        {model}
+        {input_file_name}
+        {output_file_name}
+        --score_path {score_path}
+        --task {task}
+        --num_beams 2
+        --length_penalty 2.0
+        """.split()
+
     with patch.object(sys, "argv", testargs):
         run_generate()
         assert Path(output_file_name).exists()
         os.remove(Path(output_file_name))
 
 
+# test one model to quickly (no-@slow) catch simple problems and do an
+# extensive testing of functionality with multiple models as @slow separately
+def test_run_eval():
+    run_eval_tester(T5_TINY)
+
+
+# any extra models should go into the list here - can be slow
 @slow
-@pytest.mark.parametrize("model", [pytest.param(T5_TINY)])
+@pytest.mark.parametrize("model", [BART_TINY, MBART_TINY])
+def test_run_eval_slow(model):
+    run_eval_tester(model)
+
+
+# testing with 2 models to validate: 1. translation (t5) 2. summarization (mbart)
+@slow
+@pytest.mark.parametrize("model", [T5_TINY, MBART_TINY])
 def test_run_eval_search(model):
     input_file_name = Path(tempfile.mkdtemp()) / "utest_input.source"
     output_file_name = input_file_name.parent / "utest_output.txt"
@@ -335,20 +341,17 @@ def test_run_eval_search(model):
     _dump_articles(input_file_name, text["en"])
     _dump_articles(reference_path, text["de"])
     task = "translation_en_to_de" if model == T5_TINY else "summarization"
-    testargs = [
-        "run_eval_search.py",
-        model,
-        str(input_file_name),
-        str(output_file_name),
-        "--score_path",
-        score_path,
-        "--reference_path",
-        reference_path,
-        "--task",
-        task,
-        "--search",
-        "num_beams=1:2 length_penalty=0.9:1.0",
-    ]
+    testargs = f"""
+        run_eval_search.py
+        --model_name {model}
+        --data_dir {str(input_file_name)}
+        --save_dir {str(output_file_name)}
+        --score_path {score_path}
+        --reference_path {reference_path},
+        --task {task}
+        --search num_beams=1:2 length_penalty=0.9:1.0
+        """.split()
+
     with patch.object(sys, "argv", testargs):
         with CaptureStdout() as cs:
             run_search()
@@ -367,8 +370,8 @@ def test_run_eval_search(model):
 
 
 @pytest.mark.parametrize(
-    ["model"],
-    [pytest.param(T5_TINY), pytest.param(BART_TINY), pytest.param(MBART_TINY), pytest.param(MARIAN_TINY)],
+    "model",
+    [T5_TINY, BART_TINY, MBART_TINY, MARIAN_TINY],
 )
 def test_finetune(model):
     args_d: dict = CHEAP_ARGS.copy()
@@ -520,96 +523,3 @@ def test_finetune_lr_schedulers():
     args = argparse.Namespace(**args_d1)
     model = main(args)
     assert getattr(model.hparams, "lr_scheduler") == supported_param, f"lr_scheduler={supported_param} shouldn't fail"
-
-
-def test_pack_dataset():
-    tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")
-
-    tmp_dir = Path(make_test_data_dir())
-    orig_examples = tmp_dir.joinpath("train.source").open().readlines()
-    save_dir = Path(tempfile.mkdtemp(prefix="packed_"))
-    pack_data_dir(tokenizer, tmp_dir, 128, save_dir)
-    orig_paths = {x.name for x in tmp_dir.iterdir()}
-    new_paths = {x.name for x in save_dir.iterdir()}
-    packed_examples = save_dir.joinpath("train.source").open().readlines()
-    # orig: [' Sam ate lunch today.\n', 'Sams lunch ingredients.']
-    # desired_packed: [' Sam ate lunch today.\n Sams lunch ingredients.']
-    assert len(packed_examples) < len(orig_examples)
-    assert len(packed_examples) == 1
-    assert len(packed_examples[0]) == sum(len(x) for x in orig_examples)
-    assert orig_paths == new_paths
-
-
-@pytest.mark.parametrize(
-    ["tok_name"],
-    [
-        pytest.param(MBART_TINY),
-        pytest.param(MARIAN_TINY),
-        pytest.param(T5_TINY),
-        pytest.param(BART_TINY),
-        pytest.param("google/pegasus-xsum"),
-    ],
-)
-def test_seq2seq_dataset_truncation(tok_name):
-    tokenizer = AutoTokenizer.from_pretrained(tok_name)
-    tmp_dir = make_test_data_dir()
-    max_len_source = max(len(tokenizer.encode(a)) for a in ARTICLES)
-    max_len_target = max(len(tokenizer.encode(a)) for a in SUMMARIES)
-    max_src_len = 4
-    max_tgt_len = 8
-    assert max_len_target > max_src_len  # Will be truncated
-    assert max_len_source > max_src_len  # Will be truncated
-    src_lang, tgt_lang = "ro_RO", "de_DE"  # ignored for all but mbart, but never causes error.
-    train_dataset = Seq2SeqDataset(
-        tokenizer,
-        data_dir=tmp_dir,
-        type_path="train",
-        max_source_length=max_src_len,
-        max_target_length=max_tgt_len,  # ignored
-        src_lang=src_lang,
-        tgt_lang=tgt_lang,
-    )
-    dataloader = DataLoader(train_dataset, batch_size=2, collate_fn=train_dataset.collate_fn)
-    for batch in dataloader:
-        assert isinstance(batch, dict)
-        assert batch["attention_mask"].shape == batch["input_ids"].shape
-        # show that articles were trimmed.
-        assert batch["input_ids"].shape[1] == max_src_len
-        # show that targets are the same len
-        assert batch["labels"].shape[1] == max_tgt_len
-        if tok_name != MBART_TINY:
-            continue
-        # check language codes in correct place
-        batch["decoder_input_ids"] = shift_tokens_right(batch["labels"], tokenizer.pad_token_id)
-        assert batch["decoder_input_ids"][0, 0].item() == tokenizer.lang_code_to_id[tgt_lang]
-        assert batch["decoder_input_ids"][0, -1].item() == tokenizer.eos_token_id
-        assert batch["input_ids"][0, -2].item() == tokenizer.eos_token_id
-        assert batch["input_ids"][0, -1].item() == tokenizer.lang_code_to_id[src_lang]
-
-        break  # No need to test every batch
-
-
-@pytest.mark.parametrize(["tok"], [pytest.param(BART_TINY), pytest.param("bert-base-cased")])
-def test_legacy_dataset_truncation(tok):
-    tokenizer = AutoTokenizer.from_pretrained(tok)
-    tmp_dir = make_test_data_dir()
-    max_len_source = max(len(tokenizer.encode(a)) for a in ARTICLES)
-    max_len_target = max(len(tokenizer.encode(a)) for a in SUMMARIES)
-    trunc_target = 4
-    train_dataset = LegacySeq2SeqDataset(
-        tokenizer,
-        data_dir=tmp_dir,
-        type_path="train",
-        max_source_length=20,
-        max_target_length=trunc_target,
-    )
-    dataloader = DataLoader(train_dataset, batch_size=2, collate_fn=train_dataset.collate_fn)
-    for batch in dataloader:
-        assert batch["attention_mask"].shape == batch["input_ids"].shape
-        # show that articles were trimmed.
-        assert batch["input_ids"].shape[1] == max_len_source
-        assert 20 >= batch["input_ids"].shape[1]  # trimmed significantly
-        # show that targets were truncated
-        assert batch["labels"].shape[1] == trunc_target  # Truncated
-        assert max_len_target > trunc_target  # Truncated
-        break  # No need to test every batch

examples/seq2seq/utils.py

@@ -21,6 +21,14 @@ from transformers import BartTokenizer
 from transformers.file_utils import cached_property
 
 
+try:
+    from fairseq.data.data_utils import batch_by_size
+
+    FAIRSEQ_AVAILABLE = True
+except (ImportError, ModuleNotFoundError):
+    FAIRSEQ_AVAILABLE = False
+
+
 def label_smoothed_nll_loss(lprobs, target, epsilon, ignore_index=-100):
     """From fairseq"""
     if target.dim() == lprobs.dim() - 1:
@@ -94,7 +102,13 @@ class AbstractSeq2SeqDataset(Dataset):
         super().__init__()
         self.src_file = Path(data_dir).joinpath(type_path + ".source")
         self.tgt_file = Path(data_dir).joinpath(type_path + ".target")
-        self.src_lens = self.get_char_lens(self.src_file)
+        self.len_file = Path(data_dir).joinpath(type_path + ".len")
+        if os.path.exists(self.len_file):
+            self.src_lens = pickle_load(self.len_file)
+            self.used_char_len = False
+        else:
+            self.src_lens = self.get_char_lens(self.src_file)
+            self.used_char_len = True
         self.max_source_length = max_source_length
         self.max_target_length = max_target_length
         assert min(self.src_lens) > 0, f"found empty line in {self.src_file}"
@@ -115,12 +129,42 @@ class AbstractSeq2SeqDataset(Dataset):
     def get_char_lens(data_file):
         return [len(x) for x in Path(data_file).open().readlines()]
 
+    @cached_property
+    def tgt_lens(self):
+        """Length in characters of target documents"""
+        return self.get_char_lens(self.tgt_file)
+
     def make_sortish_sampler(self, batch_size, distributed=False, shuffle=True, **kwargs):
         if distributed:
             return DistributedSortishSampler(self, batch_size, shuffle=shuffle, **kwargs)
         else:
             return SortishSampler(self.src_lens, batch_size, shuffle=shuffle)
 
+    def make_dynamic_sampler(self, max_tokens_per_batch=1024, **kwargs):
+        assert FAIRSEQ_AVAILABLE, "Dynamic batch size requires `pip install fairseq`"
+        assert not self.used_char_len, "You must call  python make_len_file.py before calling make_dynamic_sampler"
+        sorted_indices = list(self.make_sortish_sampler(1024, shuffle=False))
+
+        def num_tokens_in_example(i):
+            return min(self.src_lens[i], self.max_target_length)
+
+        # call fairseq cython function
+        batch_sampler: List[List[int]] = batch_by_size(
+            sorted_indices,
+            num_tokens_fn=num_tokens_in_example,
+            max_tokens=max_tokens_per_batch,
+            required_batch_size_multiple=64,
+        )
+        shuffled_batches = [batch_sampler[i] for i in np.random.permutation(range(len(batch_sampler)))]
+        # move the largest batch to the front to OOM quickly (uses an approximation for padding)
+        approximate_toks_per_batch = [max(self.src_lens[i] for i in batch) * len(batch) for batch in shuffled_batches]
+        largest_batch_idx = np.argmax(approximate_toks_per_batch)
+        shuffled_batches[0], shuffled_batches[largest_batch_idx] = (
+            shuffled_batches[largest_batch_idx],
+            shuffled_batches[0],
+        )
+        return shuffled_batches
+
     def __getitem__(self, item):
         raise NotImplementedError("You must implement this")
 
@@ -232,10 +276,12 @@ class DistributedSortishSampler(Sampler):
             if not dist.is_available():
                 raise RuntimeError("Requires distributed package to be available")
             num_replicas = dist.get_world_size()
+            self._rank = -1
         if rank is None:
             if not dist.is_available():
                 raise RuntimeError("Requires distributed package to be available")
             rank = dist.get_rank()
+            self._rank = rank
         self.dataset = dataset
         self.num_replicas = num_replicas
         self.rank = rank

examples/text-classification/run_glue.py

@@ -150,10 +150,11 @@ def main():
 
     def build_compute_metrics_fn(task_name: str) -> Callable[[EvalPrediction], Dict]:
         def compute_metrics_fn(p: EvalPrediction):
+            preds = p.predictions[0] if isinstance(p.predictions, tuple) else p.predictions
             if output_mode == "classification":
-                preds = np.argmax(p.predictions, axis=1)
-            elif output_mode == "regression":
-                preds = np.squeeze(p.predictions)
+                preds = np.argmax(preds, axis=1)
+            else:  # regression
+                preds = np.squeeze(preds)
             return glue_compute_metrics(task_name, preds, p.label_ids)
 
         return compute_metrics_fn

examples/token-classification/README.md

@@ -19,11 +19,11 @@ Data can be obtained from the [GermEval 2014](https://sites.google.com/site/germ
 Here are the commands for downloading and pre-processing train, dev and test datasets. The original data format has four (tab-separated) columns, in a pre-processing step only the two relevant columns (token and outer span NER annotation) are extracted:
 
 ```bash
-curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-train.tsv?attredirects=0&d=1' \
+curl -L 'https://drive.google.com/uc?export=download&id=1Jjhbal535VVz2ap4v4r_rN1UEHTdLK5P' \
 | grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > train.txt.tmp
-curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-dev.tsv?attredirects=0&d=1' \
+curl -L 'https://drive.google.com/uc?export=download&id=1ZfRcQThdtAR5PPRjIDtrVP7BtXSCUBbm' \
 | grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > dev.txt.tmp
-curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-test.tsv?attredirects=0&d=1' \
+curl -L 'https://drive.google.com/uc?export=download&id=1u9mb7kNJHWQCWyweMDRMuTFoOHOfeBTH' \
 | grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > test.txt.tmp
 ```
 

model_cards/allenai/wmt16-en-de-12-1/README.md

@@ -0,0 +1,104 @@
+
+---
+language:
+- en
+- de
+thumbnail:
+tags:
+- translation
+- wmt16
+- allenai
+license: apache-2.0
+datasets:
+- wmt16
+metrics:
+- bleu
+---
+
+# FSMT
+
+## Model description
+
+This is a ported version of fairseq-based [wmt16 transformer](https://github.com/jungokasai/deep-shallow/) for en-de.
+
+For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369).
+
+All 3 models are available:
+
+* [wmt16-en-de-dist-12-1](https://huggingface.co/allenai/wmt16-en-de-dist-12-1)
+* [wmt16-en-de-dist-6-1](https://huggingface.co/allenai/wmt16-en-de-dist-6-1)
+* [wmt16-en-de-12-1](https://huggingface.co/allenai/wmt16-en-de-12-1)
+
+
+## Intended uses & limitations
+
+#### How to use
+
+```python
+from transformers.tokenization_fsmt import FSMTTokenizer
+from transformers.modeling_fsmt import FSMTForConditionalGeneration
+mname = "allenai/wmt16-en-de-12-1"
+tokenizer = FSMTTokenizer.from_pretrained(mname)
+model = FSMTForConditionalGeneration.from_pretrained(mname)
+
+input = "Machine learning is great, isn't it?"
+input_ids = tokenizer.encode(input, return_tensors="pt")
+outputs = model.generate(input_ids)
+decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(decoded) # Maschinelles Lernen ist großartig, nicht wahr?
+
+```
+
+#### Limitations and bias
+
+
+## Training data
+
+Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369).
+
+## Eval results
+
+Here are the BLEU scores:
+
+model   | fairseq | transformers
+-------|---------|----------
+wmt16-en-de-12-1  | 26.9 | 25.75
+
+The score is slightly below the score reported in the paper, as the researchers don't use `sacrebleu` and measure the score on tokenized outputs. `transformers` score was measured using `sacrebleu` on detokenized outputs.
+
+The score was calculated using this code:
+
+```bash
+git clone https://github.com/huggingface/transformers
+cd transformers
+export PAIR=en-de
+export DATA_DIR=data/$PAIR
+export SAVE_DIR=data/$PAIR
+export BS=8
+export NUM_BEAMS=5
+mkdir -p $DATA_DIR
+sacrebleu -t wmt16 -l $PAIR --echo src > $DATA_DIR/val.source
+sacrebleu -t wmt16 -l $PAIR --echo ref > $DATA_DIR/val.target
+echo $PAIR
+PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/wmt16-en-de-12-1 $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
+```
+
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt16/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2016.tgz?1504722372)
+
+
+### BibTeX entry and citation info
+
+```
+@misc{kasai2020deep,
+    title={Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation},
+    author={Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith},
+    year={2020},
+    eprint={2006.10369},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+

model_cards/allenai/wmt16-en-de-dist-12-1/README.md

@@ -0,0 +1,104 @@
+
+---
+language:
+- en
+- de
+thumbnail:
+tags:
+- translation
+- wmt16
+- allenai
+license: apache-2.0
+datasets:
+- wmt16
+metrics:
+- bleu
+---
+
+# FSMT
+
+## Model description
+
+This is a ported version of fairseq-based [wmt16 transformer](https://github.com/jungokasai/deep-shallow/) for en-de.
+
+For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369).
+
+All 3 models are available:
+
+* [wmt16-en-de-dist-12-1](https://huggingface.co/allenai/wmt16-en-de-dist-12-1)
+* [wmt16-en-de-dist-6-1](https://huggingface.co/allenai/wmt16-en-de-dist-6-1)
+* [wmt16-en-de-12-1](https://huggingface.co/allenai/wmt16-en-de-12-1)
+
+
+## Intended uses & limitations
+
+#### How to use
+
+```python
+from transformers.tokenization_fsmt import FSMTTokenizer
+from transformers.modeling_fsmt import FSMTForConditionalGeneration
+mname = "allenai/wmt16-en-de-dist-12-1"
+tokenizer = FSMTTokenizer.from_pretrained(mname)
+model = FSMTForConditionalGeneration.from_pretrained(mname)
+
+input = "Machine learning is great, isn't it?"
+input_ids = tokenizer.encode(input, return_tensors="pt")
+outputs = model.generate(input_ids)
+decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(decoded) # Maschinelles Lernen ist großartig, nicht wahr?
+
+```
+
+#### Limitations and bias
+
+
+## Training data
+
+Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369).
+
+## Eval results
+
+Here are the BLEU scores:
+
+model   | fairseq | transformers
+-------|---------|----------
+wmt16-en-de-dist-12-1  | 28.3 | 27.52
+
+The score is slightly below the score reported in the paper, as the researchers don't use `sacrebleu` and measure the score on tokenized outputs. `transformers` score was measured using `sacrebleu` on detokenized outputs.
+
+The score was calculated using this code:
+
+```bash
+git clone https://github.com/huggingface/transformers
+cd transformers
+export PAIR=en-de
+export DATA_DIR=data/$PAIR
+export SAVE_DIR=data/$PAIR
+export BS=8
+export NUM_BEAMS=5
+mkdir -p $DATA_DIR
+sacrebleu -t wmt16 -l $PAIR --echo src > $DATA_DIR/val.source
+sacrebleu -t wmt16 -l $PAIR --echo ref > $DATA_DIR/val.target
+echo $PAIR
+PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/wmt16-en-de-dist-12-1 $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
+```
+
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt16/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2016.tgz?1504722372)
+
+
+### BibTeX entry and citation info
+
+```
+@misc{kasai2020deep,
+    title={Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation},
+    author={Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith},
+    year={2020},
+    eprint={2006.10369},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+

model_cards/allenai/wmt16-en-de-dist-6-1/README.md

@@ -0,0 +1,104 @@
+
+---
+language:
+- en
+- de
+thumbnail:
+tags:
+- translation
+- wmt16
+- allenai
+license: apache-2.0
+datasets:
+- wmt16
+metrics:
+- bleu
+---
+
+# FSMT
+
+## Model description
+
+This is a ported version of fairseq-based [wmt16 transformer](https://github.com/jungokasai/deep-shallow/) for en-de.
+
+For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369).
+
+All 3 models are available:
+
+* [wmt16-en-de-dist-12-1](https://huggingface.co/allenai/wmt16-en-de-dist-12-1)
+* [wmt16-en-de-dist-6-1](https://huggingface.co/allenai/wmt16-en-de-dist-6-1)
+* [wmt16-en-de-12-1](https://huggingface.co/allenai/wmt16-en-de-12-1)
+
+
+## Intended uses & limitations
+
+#### How to use
+
+```python
+from transformers.tokenization_fsmt import FSMTTokenizer
+from transformers.modeling_fsmt import FSMTForConditionalGeneration
+mname = "allenai/wmt16-en-de-dist-6-1"
+tokenizer = FSMTTokenizer.from_pretrained(mname)
+model = FSMTForConditionalGeneration.from_pretrained(mname)
+
+input = "Machine learning is great, isn't it?"
+input_ids = tokenizer.encode(input, return_tensors="pt")
+outputs = model.generate(input_ids)
+decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(decoded) # Maschinelles Lernen ist großartig, nicht wahr?
+
+```
+
+#### Limitations and bias
+
+
+## Training data
+
+Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369).
+
+## Eval results
+
+Here are the BLEU scores:
+
+model   | fairseq | transformers
+-------|---------|----------
+wmt16-en-de-dist-6-1  | 27.4 | 27.11
+
+The score is slightly below the score reported in the paper, as the researchers don't use `sacrebleu` and measure the score on tokenized outputs. `transformers` score was measured using `sacrebleu` on detokenized outputs.
+
+The score was calculated using this code:
+
+```bash
+git clone https://github.com/huggingface/transformers
+cd transformers
+export PAIR=en-de
+export DATA_DIR=data/$PAIR
+export SAVE_DIR=data/$PAIR
+export BS=8
+export NUM_BEAMS=5
+mkdir -p $DATA_DIR
+sacrebleu -t wmt16 -l $PAIR --echo src > $DATA_DIR/val.source
+sacrebleu -t wmt16 -l $PAIR --echo ref > $DATA_DIR/val.target
+echo $PAIR
+PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/wmt16-en-de-dist-6-1 $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
+```
+
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt16/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2016.tgz?1504722372)
+
+
+### BibTeX entry and citation info
+
+```
+@misc{kasai2020deep,
+    title={Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation},
+    author={Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith},
+    year={2020},
+    eprint={2006.10369},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+

model_cards/allenai/wmt19-de-en-6-6-base/README.md

@@ -0,0 +1,102 @@
+
+---
+
+language:
+- de
+- en
+thumbnail:
+tags:
+- translation
+- wmt19
+- allenai
+license: apache-2.0
+datasets:
+- wmt19
+metrics:
+- bleu
+---
+
+# FSMT
+
+## Model description
+
+This is a ported version of fairseq-based [wmt19 transformer](https://github.com/jungokasai/deep-shallow/) for de-en.
+
+For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369).
+
+2 models are available:
+
+* [wmt19-de-en-6-6-big](https://huggingface.co/allenai/wmt19-de-en-6-6-big)
+* [wmt19-de-en-6-6-base](https://huggingface.co/allenai/wmt19-de-en-6-6-base)
+
+
+## Intended uses & limitations
+
+#### How to use
+
+```python
+from transformers.tokenization_fsmt import FSMTTokenizer
+from transformers.modeling_fsmt import FSMTForConditionalGeneration
+mname = "allenai/wmt19-de-en-6-6-base"
+tokenizer = FSMTTokenizer.from_pretrained(mname)
+model = FSMTForConditionalGeneration.from_pretrained(mname)
+
+input = "Maschinelles Lernen ist großartig, nicht wahr?"
+input_ids = tokenizer.encode(input, return_tensors="pt")
+outputs = model.generate(input_ids)
+decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(decoded) # Machine learning is great, isn't it?
+
+```
+
+#### Limitations and bias
+
+
+## Training data
+
+Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369).
+
+## Eval results
+
+Here are the BLEU scores:
+
+model   |  transformers
+-------|---------|----------
+wmt19-de-en-6-6-base  |  38.37
+
+The score was calculated using this code:
+
+```bash
+git clone https://github.com/huggingface/transformers
+cd transformers
+export PAIR=de-en
+export DATA_DIR=data/$PAIR
+export SAVE_DIR=data/$PAIR
+export BS=8
+export NUM_BEAMS=5
+mkdir -p $DATA_DIR
+sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
+sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
+echo $PAIR
+PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/wmt19-de-en-6-6-base $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
+```
+
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt19/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)
+
+
+### BibTeX entry and citation info
+
+```
+@misc{kasai2020deep,
+    title={Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation},
+    author={Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith},
+    year={2020},
+    eprint={2006.10369},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+

model_cards/allenai/wmt19-de-en-6-6-big/README.md

@@ -0,0 +1,102 @@
+
+---
+
+language:
+- de
+- en
+thumbnail:
+tags:
+- translation
+- wmt19
+- allenai
+license: apache-2.0
+datasets:
+- wmt19
+metrics:
+- bleu
+---
+
+# FSMT
+
+## Model description
+
+This is a ported version of fairseq-based [wmt19 transformer](https://github.com/jungokasai/deep-shallow/) for de-en.
+
+For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369).
+
+2 models are available:
+
+* [wmt19-de-en-6-6-big](https://huggingface.co/allenai/wmt19-de-en-6-6-big)
+* [wmt19-de-en-6-6-base](https://huggingface.co/allenai/wmt19-de-en-6-6-base)
+
+
+## Intended uses & limitations
+
+#### How to use
+
+```python
+from transformers.tokenization_fsmt import FSMTTokenizer
+from transformers.modeling_fsmt import FSMTForConditionalGeneration
+mname = "allenai/wmt19-de-en-6-6-big"
+tokenizer = FSMTTokenizer.from_pretrained(mname)
+model = FSMTForConditionalGeneration.from_pretrained(mname)
+
+input = "Maschinelles Lernen ist großartig, nicht wahr?"
+input_ids = tokenizer.encode(input, return_tensors="pt")
+outputs = model.generate(input_ids)
+decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(decoded) # Machine learning is great, isn't it?
+
+```
+
+#### Limitations and bias
+
+
+## Training data
+
+Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369).
+
+## Eval results
+
+Here are the BLEU scores:
+
+model   |  transformers
+-------|---------|----------
+wmt19-de-en-6-6-big  |  39.9
+
+The score was calculated using this code:
+
+```bash
+git clone https://github.com/huggingface/transformers
+cd transformers
+export PAIR=de-en
+export DATA_DIR=data/$PAIR
+export SAVE_DIR=data/$PAIR
+export BS=8
+export NUM_BEAMS=5
+mkdir -p $DATA_DIR
+sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
+sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
+echo $PAIR
+PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/wmt19-de-en-6-6-big $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
+```
+
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt19/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)
+
+
+### BibTeX entry and citation info
+
+```
+@misc{kasai2020deep,
+    title={Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation},
+    author={Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith},
+    year={2020},
+    eprint={2006.10369},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+

model_cards/facebook/rag-sequence-base/README.md

@@ -0,0 +1,22 @@
+## RAG
+
+This is a "base" version of the RAG-Sequence Model of the the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/pdf/2005.11401.pdf) 
+by Patrick Lewis, Ethan Perez, Aleksandara Piktus et al.
+
+## Usage:
+
+```python
+from transformers import RagTokenizer, RagRetriever, RagSequenceForGeneration
+
+tokenizer = RagTokenizer.from_pretrained("facebook/rag-sequence-base")
+retriever = RagRetriever.from_pretrained("facebook/rag-sequence-base", index_name="exact", use_dummy_dataset=True)
+model = RagSequenceForGeneration.from_pretrained("facebook/rag-sequence-base", retriever=retriever)
+
+input_ids = tokenizer("What is the largest country in the world?", return_tensors="pt").input_ids
+
+generated = model.generate(input_ids=input_ids)
+generated_string = tokenizer.batch_decode(generated, skip_special_tokens=True)
+
+# => should give ["Asia ended in 2010 when China overtook Japan to become the world's second largest economy."]
+# Interesting answer. Definitely on topic, but might factual probably not fully correct.
+```

model_cards/facebook/rag-sequence-nq/README.md

@@ -0,0 +1,25 @@
+## RAG
+
+This is the RAG-Sequence Model of the the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/pdf/2005.11401.pdf) 
+by Patrick Lewis, Ethan Perez, Aleksandara Piktus et al.
+
+## Usage:
+
+```python
+
+from transformers import RagTokenizer, RagRetriever, RagSequenceForGeneration
+
+tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")
+retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True)
+model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
+
+input_dict = tokenizer.prepare_seq2seq_batch("How many people live in Paris?", "In Paris, there are 10 million people.", return_tensors="pt")
+outputs = model(input_ids=input_dict["input_ids"], labels=input_dict["labels"])
+
+# outputs.loss should give 76.2978
+
+generated = model.generate(input_ids=input_dict["input_ids"])
+generated_string = tokenizer.batch_decode(generated, skip_special_tokens=True)
+
+# generated_string should give 270,000,000 -> not quite correct the answer, but it also only uses a dummy index
+```

model_cards/facebook/rag-token-base/README.md

@@ -0,0 +1,21 @@
+## RAG
+
+This is a "base" version of the RAG-Token Model of the the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/pdf/2005.11401.pdf) 
+by Patrick Lewis, Ethan Perez, Aleksandara Piktus et al.
+
+## Usage:
+
+```python
+from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration
+
+tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-base")
+retriever = RagRetriever.from_pretrained("facebook/rag-token-base", index_name="exact", use_dummy_dataset=True)
+model = RagTokenForGeneration.from_pretrained("facebook/rag-token-base", retriever=retriever)
+
+input_ids = tokenizer("What is the largest country in the world?", return_tensors="pt").input_ids
+
+generated = model.generate(input_ids=input_ids)
+generated_string = tokenizer.batch_decode(generated, skip_special_tokens=True)
+
+# => should give [' russia']. Pretty good answer for just having just a dummy dataset.
+```

model_cards/facebook/rag-token-nq/README.md

@@ -0,0 +1,25 @@
+## RAG
+
+This is the RAG-Token Model of the the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/pdf/2005.11401.pdf) 
+by Patrick Lewis, Ethan Perez, Aleksandara Piktus et al.
+
+## Usage:
+
+```python
+
+from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration
+
+tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")
+retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True)
+model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
+
+input_dict = tokenizer.prepare_seq2seq_batch("How many people live in Paris?", "In Paris, there are 10 million people.", return_tensors="pt")
+outputs = model(input_ids=input_dict["input_ids"], labels=input_dict["labels"])
+
+# outputs.loss should give 76.1230
+
+generated = model.generate(input_ids=input_dict["input_ids"])
+generated_string = tokenizer.batch_decode(generated, skip_special_tokens=True)
+
+# generated_string should give 270,000 -> not quite correct the answer, but it also only uses a dummy index
+```

model_cards/facebook/wmt19-de-en/README.md

@@ -0,0 +1,111 @@
+
+---
+language: 
+- de
+- en
+thumbnail:
+tags:
+- translation
+- wmt19
+- facebook
+license: apache-2.0
+datasets:
+- wmt19
+metrics:
+- bleu
+---
+
+# FSMT
+
+## Model description
+
+This is a ported version of [fairseq wmt19 transformer](https://github.com/pytorch/fairseq/blob/master/examples/wmt19/README.md) for de-en.
+
+For more details, please see, [Facebook FAIR's WMT19 News Translation Task Submission](https://arxiv.org/abs/1907.06616).
+
+The abbreviation FSMT stands for FairSeqMachineTranslation
+
+All four models are available:
+
+* [wmt19-en-ru](https://huggingface.co/facebook/wmt19-en-ru)
+* [wmt19-ru-en](https://huggingface.co/facebook/wmt19-ru-en)
+* [wmt19-en-de](https://huggingface.co/facebook/wmt19-en-de)
+* [wmt19-de-en](https://huggingface.co/facebook/wmt19-de-en)
+
+## Intended uses & limitations
+
+#### How to use
+
+```python
+from transformers.tokenization_fsmt import FSMTTokenizer
+from transformers.modeling_fsmt import FSMTForConditionalGeneration
+mname = "facebook/wmt19-de-en"
+tokenizer = FSMTTokenizer.from_pretrained(mname)
+model = FSMTForConditionalGeneration.from_pretrained(mname)
+
+input = "Maschinelles Lernen ist großartig, oder?"
+input_ids = tokenizer.encode(input, return_tensors="pt")
+outputs = model.generate(input_ids)
+decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(decoded) # Machine learning is great, isn't it?
+
+```
+
+#### Limitations and bias
+
+- The original (and this ported model) doesn't seem to handle well inputs with repeated sub-phrases, [content gets truncated](https://discuss.huggingface.co/t/issues-with-translating-inputs-containing-repeated-phrases/981)
+
+## Training data
+
+Pretrained weights were left identical to the original model released by fairseq. For more details, please, see the [paper](https://arxiv.org/abs/1907.06616).
+
+## Eval results
+
+pair   | fairseq | transformers
+-------|---------|----------
+de-en  | [42.3](http://matrix.statmt.org/matrix/output/1902?run_id=6750) | 41.35
+
+The score is slightly below the score reported by `fairseq`, since `transformers`` currently doesn't support:
+- model ensemble, therefore the best performing checkpoint was ported (``model4.pt``).
+- re-ranking
+
+The score was calculated using this code:
+
+```bash
+git clone https://github.com/huggingface/transformers
+cd transformers
+export PAIR=de-en
+export DATA_DIR=data/$PAIR
+export SAVE_DIR=data/$PAIR
+export BS=8
+export NUM_BEAMS=15
+mkdir -p $DATA_DIR
+sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
+sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
+echo $PAIR
+PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py facebook/wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
+```
+note: fairseq reports using a beam of 50, so you should get a slightly higher score if re-run with `--num_beams 50`.
+
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt19/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@inproceedings{...,
+  year={2020},
+  title={Facebook FAIR's WMT19 News Translation Task Submission},
+  author={Ng, Nathan and Yee, Kyra and Baevski, Alexei and Ott, Myle and Auli, Michael and Edunov, Sergey},
+  booktitle={Proc. of WMT},
+}
+```
+
+
+## TODO
+
+- port model ensemble (fairseq uses 4 model checkpoints)
+

model_cards/facebook/wmt19-en-de/README.md

@@ -0,0 +1,111 @@
+
+---
+language: 
+- en
+- de
+thumbnail:
+tags:
+- translation
+- wmt19
+- facebook
+license: apache-2.0
+datasets:
+- wmt19
+metrics:
+- bleu
+---
+
+# FSMT
+
+## Model description
+
+This is a ported version of [fairseq wmt19 transformer](https://github.com/pytorch/fairseq/blob/master/examples/wmt19/README.md) for en-de.
+
+For more details, please see, [Facebook FAIR's WMT19 News Translation Task Submission](https://arxiv.org/abs/1907.06616).
+
+The abbreviation FSMT stands for FairSeqMachineTranslation
+
+All four models are available:
+
+* [wmt19-en-ru](https://huggingface.co/facebook/wmt19-en-ru)
+* [wmt19-ru-en](https://huggingface.co/facebook/wmt19-ru-en)
+* [wmt19-en-de](https://huggingface.co/facebook/wmt19-en-de)
+* [wmt19-de-en](https://huggingface.co/facebook/wmt19-de-en)
+
+## Intended uses & limitations
+
+#### How to use
+
+```python
+from transformers.tokenization_fsmt import FSMTTokenizer
+from transformers.modeling_fsmt import FSMTForConditionalGeneration
+mname = "facebook/wmt19-en-de"
+tokenizer = FSMTTokenizer.from_pretrained(mname)
+model = FSMTForConditionalGeneration.from_pretrained(mname)
+
+input = "Machine learning is great, isn't it?"
+input_ids = tokenizer.encode(input, return_tensors="pt")
+outputs = model.generate(input_ids)
+decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(decoded) # Maschinelles Lernen ist großartig, oder?
+
+```
+
+#### Limitations and bias
+
+- The original (and this ported model) doesn't seem to handle well inputs with repeated sub-phrases, [content gets truncated](https://discuss.huggingface.co/t/issues-with-translating-inputs-containing-repeated-phrases/981)
+
+## Training data
+
+Pretrained weights were left identical to the original model released by fairseq. For more details, please, see the [paper](https://arxiv.org/abs/1907.06616).
+
+## Eval results
+
+pair   | fairseq | transformers
+-------|---------|----------
+en-de  | [43.1](http://matrix.statmt.org/matrix/output/1909?run_id=6862) | 42.83
+
+The score is slightly below the score reported by `fairseq`, since `transformers`` currently doesn't support:
+- model ensemble, therefore the best performing checkpoint was ported (``model4.pt``).
+- re-ranking
+
+The score was calculated using this code:
+
+```bash
+git clone https://github.com/huggingface/transformers
+cd transformers
+export PAIR=en-de
+export DATA_DIR=data/$PAIR
+export SAVE_DIR=data/$PAIR
+export BS=8
+export NUM_BEAMS=15
+mkdir -p $DATA_DIR
+sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
+sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
+echo $PAIR
+PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py facebook/wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
+```
+note: fairseq reports using a beam of 50, so you should get a slightly higher score if re-run with `--num_beams 50`.
+
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt19/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@inproceedings{...,
+  year={2020},
+  title={Facebook FAIR's WMT19 News Translation Task Submission},
+  author={Ng, Nathan and Yee, Kyra and Baevski, Alexei and Ott, Myle and Auli, Michael and Edunov, Sergey},
+  booktitle={Proc. of WMT},
+}
+```
+
+
+## TODO
+
+- port model ensemble (fairseq uses 4 model checkpoints)
+

model_cards/facebook/wmt19-en-ru/README.md

@@ -0,0 +1,111 @@
+
+---
+language: 
+- en
+- ru
+thumbnail:
+tags:
+- translation
+- wmt19
+- facebook
+license: apache-2.0
+datasets:
+- wmt19
+metrics:
+- bleu
+---
+
+# FSMT
+
+## Model description
+
+This is a ported version of [fairseq wmt19 transformer](https://github.com/pytorch/fairseq/blob/master/examples/wmt19/README.md) for en-ru.
+
+For more details, please see, [Facebook FAIR's WMT19 News Translation Task Submission](https://arxiv.org/abs/1907.06616).
+
+The abbreviation FSMT stands for FairSeqMachineTranslation
+
+All four models are available:
+
+* [wmt19-en-ru](https://huggingface.co/facebook/wmt19-en-ru)
+* [wmt19-ru-en](https://huggingface.co/facebook/wmt19-ru-en)
+* [wmt19-en-de](https://huggingface.co/facebook/wmt19-en-de)
+* [wmt19-de-en](https://huggingface.co/facebook/wmt19-de-en)
+
+## Intended uses & limitations
+
+#### How to use
+
+```python
+from transformers.tokenization_fsmt import FSMTTokenizer
+from transformers.modeling_fsmt import FSMTForConditionalGeneration
+mname = "facebook/wmt19-en-ru"
+tokenizer = FSMTTokenizer.from_pretrained(mname)
+model = FSMTForConditionalGeneration.from_pretrained(mname)
+
+input = "Machine learning is great, isn't it?"
+input_ids = tokenizer.encode(input, return_tensors="pt")
+outputs = model.generate(input_ids)
+decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(decoded) # Машинное обучение - это здорово, не так ли?
+
+```
+
+#### Limitations and bias
+
+- The original (and this ported model) doesn't seem to handle well inputs with repeated sub-phrases, [content gets truncated](https://discuss.huggingface.co/t/issues-with-translating-inputs-containing-repeated-phrases/981)
+
+## Training data
+
+Pretrained weights were left identical to the original model released by fairseq. For more details, please, see the [paper](https://arxiv.org/abs/1907.06616).
+
+## Eval results
+
+pair   | fairseq | transformers
+-------|---------|----------
+en-ru  | [36.4](http://matrix.statmt.org/matrix/output/1914?run_id=6724) | 33.47
+
+The score is slightly below the score reported by `fairseq`, since `transformers`` currently doesn't support:
+- model ensemble, therefore the best performing checkpoint was ported (``model4.pt``).
+- re-ranking
+
+The score was calculated using this code:
+
+```bash
+git clone https://github.com/huggingface/transformers
+cd transformers
+export PAIR=en-ru
+export DATA_DIR=data/$PAIR
+export SAVE_DIR=data/$PAIR
+export BS=8
+export NUM_BEAMS=15
+mkdir -p $DATA_DIR
+sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
+sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
+echo $PAIR
+PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py facebook/wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
+```
+note: fairseq reports using a beam of 50, so you should get a slightly higher score if re-run with `--num_beams 50`.
+
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt19/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@inproceedings{...,
+  year={2020},
+  title={Facebook FAIR's WMT19 News Translation Task Submission},
+  author={Ng, Nathan and Yee, Kyra and Baevski, Alexei and Ott, Myle and Auli, Michael and Edunov, Sergey},
+  booktitle={Proc. of WMT},
+}
+```
+
+
+## TODO
+
+- port model ensemble (fairseq uses 4 model checkpoints)
+

model_cards/facebook/wmt19-ru-en/README.md

@@ -0,0 +1,111 @@
+
+---
+language: 
+- ru
+- en
+thumbnail:
+tags:
+- translation
+- wmt19
+- facebook
+license: apache-2.0
+datasets:
+- wmt19
+metrics:
+- bleu
+---
+
+# FSMT
+
+## Model description
+
+This is a ported version of [fairseq wmt19 transformer](https://github.com/pytorch/fairseq/blob/master/examples/wmt19/README.md) for ru-en.
+
+For more details, please see, [Facebook FAIR's WMT19 News Translation Task Submission](https://arxiv.org/abs/1907.06616).
+
+The abbreviation FSMT stands for FairSeqMachineTranslation
+
+All four models are available:
+
+* [wmt19-en-ru](https://huggingface.co/facebook/wmt19-en-ru)
+* [wmt19-ru-en](https://huggingface.co/facebook/wmt19-ru-en)
+* [wmt19-en-de](https://huggingface.co/facebook/wmt19-en-de)
+* [wmt19-de-en](https://huggingface.co/facebook/wmt19-de-en)
+
+## Intended uses & limitations
+
+#### How to use
+
+```python
+from transformers.tokenization_fsmt import FSMTTokenizer
+from transformers.modeling_fsmt import FSMTForConditionalGeneration
+mname = "facebook/wmt19-ru-en"
+tokenizer = FSMTTokenizer.from_pretrained(mname)
+model = FSMTForConditionalGeneration.from_pretrained(mname)
+
+input = "Машинное обучение - это здорово, не так ли?"
+input_ids = tokenizer.encode(input, return_tensors="pt")
+outputs = model.generate(input_ids)
+decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(decoded) # Machine learning is great, isn't it?
+
+```
+
+#### Limitations and bias
+
+- The original (and this ported model) doesn't seem to handle well inputs with repeated sub-phrases, [content gets truncated](https://discuss.huggingface.co/t/issues-with-translating-inputs-containing-repeated-phrases/981)
+
+## Training data
+
+Pretrained weights were left identical to the original model released by fairseq. For more details, please, see the [paper](https://arxiv.org/abs/1907.06616).
+
+## Eval results
+
+pair   | fairseq | transformers
+-------|---------|----------
+ru-en  | [41.3](http://matrix.statmt.org/matrix/output/1907?run_id=6937) | 39.20
+
+The score is slightly below the score reported by `fairseq`, since `transformers`` currently doesn't support:
+- model ensemble, therefore the best performing checkpoint was ported (``model4.pt``).
+- re-ranking
+
+The score was calculated using this code:
+
+```bash
+git clone https://github.com/huggingface/transformers
+cd transformers
+export PAIR=ru-en
+export DATA_DIR=data/$PAIR
+export SAVE_DIR=data/$PAIR
+export BS=8
+export NUM_BEAMS=15
+mkdir -p $DATA_DIR
+sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
+sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
+echo $PAIR
+PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py facebook/wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
+```
+note: fairseq reports using a beam of 50, so you should get a slightly higher score if re-run with `--num_beams 50`.
+
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt19/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@inproceedings{...,
+  year={2020},
+  title={Facebook FAIR's WMT19 News Translation Task Submission},
+  author={Ng, Nathan and Yee, Kyra and Baevski, Alexei and Ott, Myle and Auli, Michael and Edunov, Sergey},
+  booktitle={Proc. of WMT},
+}
+```
+
+
+## TODO
+
+- port model ensemble (fairseq uses 4 model checkpoints)
+

model_cards/indobenchmark/indobert-base-p1/README.md

@@ -0,0 +1,60 @@
+---
+language: id
+tags:
+- indobert
+- indobenchmark
+- indonlu
+license: mit
+inference: false
+datasets:
+- Indo4B
+---
+
+# IndoBERT Base Model (phase1 - uncased)
+
+[IndoBERT](https://arxiv.org/abs/2009.05387) is a state-of-the-art language model for Indonesian based on the BERT model. The pretrained model is trained using a masked language modeling (MLM) objective and next sentence prediction (NSP) objective. 
+
+## All Pre-trained Models
+
+| Model                          | #params                        | Arch. | Training data                     |
+|--------------------------------|--------------------------------|-------|-----------------------------------|
+| `indobenchmark/indobert-base-p1` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-base-p2` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p1` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p2` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p1` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p2` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p1` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p2` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+
+## How to use
+
+### Load model and tokenizer
+```python
+from transformers import BertTokenizer, AutoModel
+tokenizer = BertTokenizer.from_pretrained("indobenchmark/indobert-base-p1")
+model = AutoModel.from_pretrained("indobenchmark/indobert-base-p1")
+```
+
+### Extract contextual representation
+```python
+x = torch.LongTensor(tokenizer.encode('aku adalah anak [MASK]')).view(1,-1)
+print(x, model(x)[0].sum())
+```
+
+## Authors 
+
+<b>IndoBERT</b> was trained and evaluated by Bryan Wilie\*, Karissa Vincentio\*, Genta Indra Winata\*, Samuel Cahyawijaya\*, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, Ayu Purwarianti.
+
+
+## Citation
+If you use our work, please cite:
+
+```bibtex
+@inproceedings{wilie2020indonlu,
+  title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
+  author={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},
+  booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
+  year={2020}
+}
+```

model_cards/indobenchmark/indobert-base-p2/README.md

@@ -0,0 +1,60 @@
+---
+language: id
+tags:
+- indobert
+- indobenchmark
+- indonlu
+license: mit
+inference: false
+datasets:
+- Indo4B
+---
+
+# IndoBERT Base Model (phase2 - uncased)
+
+[IndoBERT](https://arxiv.org/abs/2009.05387) is a state-of-the-art language model for Indonesian based on the BERT model. The pretrained model is trained using a masked language modeling (MLM) objective and next sentence prediction (NSP) objective. 
+
+## All Pre-trained Models
+
+| Model                          | #params                        | Arch. | Training data                     |
+|--------------------------------|--------------------------------|-------|-----------------------------------|
+| `indobenchmark/indobert-base-p1` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-base-p2` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p1` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p2` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p1` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p2` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p1` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p2` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+
+## How to use
+
+### Load model and tokenizer
+```python
+from transformers import BertTokenizer, AutoModel
+tokenizer = BertTokenizer.from_pretrained("indobenchmark/indobert-base-p2")
+model = AutoModel.from_pretrained("indobenchmark/indobert-base-p2")
+```
+
+### Extract contextual representation
+```python
+x = torch.LongTensor(tokenizer.encode('aku adalah anak [MASK]')).view(1,-1)
+print(x, model(x)[0].sum())
+```
+
+## Authors 
+
+<b>IndoBERT</b> was trained and evaluated by Bryan Wilie\*, Karissa Vincentio\*, Genta Indra Winata\*, Samuel Cahyawijaya\*, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, Ayu Purwarianti.
+
+
+## Citation
+If you use our work, please cite:
+
+```bibtex
+@inproceedings{wilie2020indonlu,
+  title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
+  author={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},
+  booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
+  year={2020}
+}
+```

model_cards/indobenchmark/indobert-large-p1/README.md

@@ -0,0 +1,60 @@
+---
+language: id
+tags:
+- indobert
+- indobenchmark
+- indonlu
+license: mit
+inference: false
+datasets:
+- Indo4B
+---
+
+# IndoBERT Large Model (phase1 - uncased)
+
+[IndoBERT](https://arxiv.org/abs/2009.05387) is a state-of-the-art language model for Indonesian based on the BERT model. The pretrained model is trained using a masked language modeling (MLM) objective and next sentence prediction (NSP) objective. 
+
+## All Pre-trained Models
+
+| Model                          | #params                        | Arch. | Training data                     |
+|--------------------------------|--------------------------------|-------|-----------------------------------|
+| `indobenchmark/indobert-base-p1` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-base-p2` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p1` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p2` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p1` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p2` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p1` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p2` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+
+## How to use
+
+### Load model and tokenizer
+```python
+from transformers import BertTokenizer, AutoModel
+tokenizer = BertTokenizer.from_pretrained("indobenchmark/indobert-large-p1")
+model = AutoModel.from_pretrained("indobenchmark/indobert-large-p1")
+```
+
+### Extract contextual representation
+```python
+x = torch.LongTensor(tokenizer.encode('aku adalah anak [MASK]')).view(1,-1)
+print(x, model(x)[0].sum())
+```
+
+## Authors 
+
+<b>IndoBERT</b> was trained and evaluated by Bryan Wilie\*, Karissa Vincentio\*, Genta Indra Winata\*, Samuel Cahyawijaya\*, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, Ayu Purwarianti.
+
+
+## Citation
+If you use our work, please cite:
+
+```bibtex
+@inproceedings{wilie2020indonlu,
+  title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
+  author={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},
+  booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
+  year={2020}
+}
+```

model_cards/indobenchmark/indobert-large-p2/README.md

@@ -0,0 +1,60 @@
+---
+language: id
+tags:
+- indobert
+- indobenchmark
+- indonlu
+license: mit
+inference: false
+datasets:
+- Indo4B
+---
+
+# IndoBERT Large Model (phase2 - uncased)
+
+[IndoBERT](https://arxiv.org/abs/2009.05387) is a state-of-the-art language model for Indonesian based on the BERT model. The pretrained model is trained using a masked language modeling (MLM) objective and next sentence prediction (NSP) objective. 
+
+## All Pre-trained Models
+
+| Model                          | #params                        | Arch. | Training data                     |
+|--------------------------------|--------------------------------|-------|-----------------------------------|
+| `indobenchmark/indobert-base-p1` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-base-p2` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p1` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p2` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p1` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p2` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p1` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p2` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+
+## How to use
+
+### Load model and tokenizer
+```python
+from transformers import BertTokenizer, AutoModel
+tokenizer = BertTokenizer.from_pretrained("indobenchmark/indobert-large-p2")
+model = AutoModel.from_pretrained("indobenchmark/indobert-large-p2")
+```
+
+### Extract contextual representation
+```python
+x = torch.LongTensor(tokenizer.encode('aku adalah anak [MASK]')).view(1,-1)
+print(x, model(x)[0].sum())
+```
+
+## Authors 
+
+<b>IndoBERT</b> was trained and evaluated by Bryan Wilie\*, Karissa Vincentio\*, Genta Indra Winata\*, Samuel Cahyawijaya\*, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, Ayu Purwarianti.
+
+
+## Citation
+If you use our work, please cite:
+
+```bibtex
+@inproceedings{wilie2020indonlu,
+  title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
+  author={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},
+  booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
+  year={2020}
+}
+```

model_cards/indobenchmark/indobert-lite-base-p1/README.md

@@ -0,0 +1,60 @@
+---
+language: id
+tags:
+- indobert
+- indobenchmark
+- indonlu
+license: mit
+inference: false
+datasets:
+- Indo4B
+---
+
+# IndoBERT-Lite Base Model (phase1 - uncased)
+
+[IndoBERT](https://arxiv.org/abs/2009.05387) is a state-of-the-art language model for Indonesian based on the BERT model. The pretrained model is trained using a masked language modeling (MLM) objective and next sentence prediction (NSP) objective. 
+
+## All Pre-trained Models
+
+| Model                          | #params                        | Arch. | Training data                     |
+|--------------------------------|--------------------------------|-------|-----------------------------------|
+| `indobenchmark/indobert-base-p1` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-base-p2` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p1` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p2` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p1` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p2` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p1` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p2` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+
+## How to use
+
+### Load model and tokenizer
+```python
+from transformers import BertTokenizer, AutoModel
+tokenizer = BertTokenizer.from_pretrained("indobenchmark/indobert-lite-base-p1")
+model = AutoModel.from_pretrained("indobenchmark/indobert-lite-base-p1")
+```
+
+### Extract contextual representation
+```python
+x = torch.LongTensor(tokenizer.encode('aku adalah anak [MASK]')).view(1,-1)
+print(x, model(x)[0].sum())
+```
+
+## Authors 
+
+<b>IndoBERT</b> was trained and evaluated by Bryan Wilie\*, Karissa Vincentio\*, Genta Indra Winata\*, Samuel Cahyawijaya\*, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, Ayu Purwarianti.
+
+
+## Citation
+If you use our work, please cite:
+
+```bibtex
+@inproceedings{wilie2020indonlu,
+  title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
+  author={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},
+  booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
+  year={2020}
+}
+```

model_cards/indobenchmark/indobert-lite-base-p2/README.md

@@ -0,0 +1,60 @@
+---
+language: id
+tags:
+- indobert
+- indobenchmark
+- indonlu
+license: mit
+inference: false
+datasets:
+- Indo4B
+---
+
+# IndoBERT-Lite Base Model (phase2 - uncased)
+
+[IndoBERT](https://arxiv.org/abs/2009.05387) is a state-of-the-art language model for Indonesian based on the BERT model. The pretrained model is trained using a masked language modeling (MLM) objective and next sentence prediction (NSP) objective. 
+
+## All Pre-trained Models
+
+| Model                          | #params                        | Arch. | Training data                     |
+|--------------------------------|--------------------------------|-------|-----------------------------------|
+| `indobenchmark/indobert-base-p1` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-base-p2` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p1` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p2` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p1` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p2` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p1` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p2` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+
+## How to use
+
+### Load model and tokenizer
+```python
+from transformers import BertTokenizer, AutoModel
+tokenizer = BertTokenizer.from_pretrained("indobenchmark/indobert-lite-base-p2")
+model = AutoModel.from_pretrained("indobenchmark/indobert-lite-base-p2")
+```
+
+### Extract contextual representation
+```python
+x = torch.LongTensor(tokenizer.encode('aku adalah anak [MASK]')).view(1,-1)
+print(x, model(x)[0].sum())
+```
+
+## Authors 
+
+<b>IndoBERT</b> was trained and evaluated by Bryan Wilie\*, Karissa Vincentio\*, Genta Indra Winata\*, Samuel Cahyawijaya\*, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, Ayu Purwarianti.
+
+
+## Citation
+If you use our work, please cite:
+
+```bibtex
+@inproceedings{wilie2020indonlu,
+  title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
+  author={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},
+  booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
+  year={2020}
+}
+```

model_cards/indobenchmark/indobert-lite-large-p1/README.md

@@ -0,0 +1,60 @@
+---
+language: id
+tags:
+- indobert
+- indobenchmark
+- indonlu
+license: mit
+inference: false
+datasets:
+- Indo4B
+---
+
+# IndoBERT-Lite Large Model (phase1 - uncased)
+
+[IndoBERT](https://arxiv.org/abs/2009.05387) is a state-of-the-art language model for Indonesian based on the BERT model. The pretrained model is trained using a masked language modeling (MLM) objective and next sentence prediction (NSP) objective. 
+
+## All Pre-trained Models
+
+| Model                          | #params                        | Arch. | Training data                     |
+|--------------------------------|--------------------------------|-------|-----------------------------------|
+| `indobenchmark/indobert-base-p1` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-base-p2` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p1` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p2` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p1` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p2` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p1` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p2` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+
+## How to use
+
+### Load model and tokenizer
+```python
+from transformers import BertTokenizer, AutoModel
+tokenizer = BertTokenizer.from_pretrained("indobenchmark/indobert-lite-large-p1")
+model = AutoModel.from_pretrained("indobenchmark/indobert-lite-large-p1")
+```
+
+### Extract contextual representation
+```python
+x = torch.LongTensor(tokenizer.encode('aku adalah anak [MASK]')).view(1,-1)
+print(x, model(x)[0].sum())
+```
+
+## Authors 
+
+<b>IndoBERT</b> was trained and evaluated by Bryan Wilie\*, Karissa Vincentio\*, Genta Indra Winata\*, Samuel Cahyawijaya\*, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, Ayu Purwarianti.
+
+
+## Citation
+If you use our work, please cite:
+
+```bibtex
+@inproceedings{wilie2020indonlu,
+  title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
+  author={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},
+  booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
+  year={2020}
+}
+```

model_cards/indobenchmark/indobert-lite-large-p2/README.md

@@ -0,0 +1,60 @@
+---
+language: id
+tags:
+- indobert
+- indobenchmark
+- indonlu
+license: mit
+inference: false
+datasets:
+- Indo4B
+---
+
+# IndoBERT-Lite Large Model (phase2 - uncased)
+
+[IndoBERT](https://arxiv.org/abs/2009.05387) is a state-of-the-art language model for Indonesian based on the BERT model. The pretrained model is trained using a masked language modeling (MLM) objective and next sentence prediction (NSP) objective. 
+
+## All Pre-trained Models
+
+| Model                          | #params                        | Arch. | Training data                     |
+|--------------------------------|--------------------------------|-------|-----------------------------------|
+| `indobenchmark/indobert-base-p1` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-base-p2` | 124.5M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p1` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-large-p2` | 335.2M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p1` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-base-p2` | 11.7M   | Base  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p1` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+| `indobenchmark/indobert-lite-large-p2` | 17.7M   | Large  | Indo4B (23.43 GB of text)            |
+
+## How to use
+
+### Load model and tokenizer
+```python
+from transformers import BertTokenizer, AutoModel
+tokenizer = BertTokenizer.from_pretrained("indobenchmark/indobert-lite-large-p2")
+model = AutoModel.from_pretrained("indobenchmark/indobert-lite-large-p2")
+```
+
+### Extract contextual representation
+```python
+x = torch.LongTensor(tokenizer.encode('aku adalah anak [MASK]')).view(1,-1)
+print(x, model(x)[0].sum())
+```
+
+## Authors 
+
+<b>IndoBERT</b> was trained and evaluated by Bryan Wilie\*, Karissa Vincentio\*, Genta Indra Winata\*, Samuel Cahyawijaya\*, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, Ayu Purwarianti.
+
+
+## Citation
+If you use our work, please cite:
+
+```bibtex
+@inproceedings{wilie2020indonlu,
+  title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
+  author={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},
+  booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
+  year={2020}
+}
+```

model_cards/mrm8488/GuaPeTe-2-tiny/README.md

@@ -0,0 +1,7 @@
+---
+language: es
+widget:
+- text: "Murcia es la huerta de Europa porque"
+---
+
+#GuaPeTe-2-tiny: A proof of concept tiny GPT-2 like model trained on Spanish Wikipedia corpus

model_cards/mrm8488/RuPERTa-base-finetuned-pawsx-es/README.md

@@ -0,0 +1,9 @@
+---
+language: es
+datasets:
+- xtreme
+widget:
+- text: "En 2009 se mudó a Filadelfia y en la actualidad vive en Nueva York. Se mudó nuevamente a Filadelfia en 2009 y ahora vive en la ciudad de Nueva York."
+---
+
+# RuPERTa-base fine-tuned on PAWS-X-es for Paraphrase Identification

model_cards/mrm8488/bert-base-german-dbmdz-cased-finetuned-pawsx-de/README.md

@@ -0,0 +1,9 @@
+---
+language: de
+datasets:
+- xtreme
+widget:
+- text: "Winarsky ist Mitglied des IEEE, Phi Beta Kappa, des ACM und des Sigma Xi. Winarsky ist Mitglied des ACM, des IEEE, der Phi Beta Kappa und der Sigma Xi."
+---
+
+# bert-base-german-dbmdz-cased fine-tuned on PAWS-X-de for Paraphrase Identification

model_cards/mrm8488/camembert-base-finetuned-pawsx-fr/README.md

@@ -0,0 +1,9 @@
+---
+language: fr
+datasets:
+- xtreme
+widget:
+- text: "La première série a été mieux reçue par la critique que la seconde. La seconde série a été bien accueillie par la critique, mieux que la première."
+---
+
+# Camembert-base fine-tuned on PAWS-X-fr for Paraphrase Identification

model_cards/mrm8488/electricidad-base-finetuned-pawsx-es/README.md

@@ -0,0 +1,9 @@
+---
+language: es
+datasets:
+- xtreme
+widget:
+- text: "El río Tabaci es una vertiente del río Leurda en Rumania. El río Leurda es un afluente del río Tabaci en Rumania."
+---
+
+# Electricidad-base fine-tuned on PAWS-X-es for Paraphrase Identification

model_cards/mrm8488/electricidad-base-generator/README.md

@@ -1,6 +1,8 @@
 ---
 language: es
 thumbnail: https://i.imgur.com/uxAvBfh.png
+widget:
+- text: "Madrid es una ciudad muy [MASK] en España."
 
 
 ---

model_cards/stas/tiny-wmt19-en-de/README.md

@@ -0,0 +1,18 @@
+---
+language:
+- en
+- de
+thumbnail:
+tags:
+- wmt19
+- testing
+license: apache-2.0
+datasets:
+- wmt19
+metrics:
+- bleu
+---
+
+# Tiny FSMT
+
+This is a tiny model that is used in the `transformers` test suite. It doesn't do anything useful, other than testing that `FSMT` works.

model_cards/tuner007/pegasus_qa/README.md

@@ -0,0 +1,30 @@
+# Pegasus for question-answering
+Pegasus model fine-tuned for QA using text-to-text approach
+
+## Model in Action 🚀
+```
+import torch
+from transformers import PegasusForConditionalGeneration, PegasusTokenizer
+model_name = 'tuner007/pegasus_qa'
+torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'
+tokenizer = PegasusTokenizer.from_pretrained(model_name)
+model = PegasusForConditionalGeneration.from_pretrained(model_name).to(torch_device)
+
+def get_answer(question, context):
+  input_text = "question: %s text: %s" % (question,context)
+  batch = tokenizer.prepare_seq2seq_batch([input_text], truncation=True, padding='longest').to(torch_device)
+  translated = model.generate(**batch)
+  tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)
+  return tgt_text[0]
+```
+#### Example:
+```
+context = "PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow."
+question = "How many customers were affected by the shutoffs?"
+get_answer(question, context)
+# output: '800 thousand'
+```
+
+
+> Created by Arpit Rajauria
+[![Twitter icon](https://cdn0.iconfinder.com/data/icons/shift-logotypes/32/Twitter-32.png)](https://twitter.com/arpit_rajauria)

model_cards/valhalla/distilbart-mnli-12-1/README.md

@@ -0,0 +1,58 @@
+---
+datasets:
+- mnli
+tags:
+- distilbart
+- distilbart-mnli
+---
+
+# DistilBart-MNLI
+
+distilbart-mnli is the distilled version of bart-large-mnli created using the **No Teacher Distillation** technique proposed for BART summarisation by Huggingface, [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq#distilbart).
+
+We just copy alternating layers from `bart-large-mnli` and finetune more on the same data. 
+
+
+|                                                                                      | matched acc | mismatched acc |
+| ------------------------------------------------------------------------------------ | ----------- | -------------- |
+| [bart-large-mnli](https://huggingface.co/facebook/bart-large-mnli) (baseline, 12-12) | 89.9        | 90.01          |
+| [distilbart-mnli-12-1](https://huggingface.co/valhalla/distilbart-mnli-12-1)         | 87.08       | 87.5           |
+| [distilbart-mnli-12-3](https://huggingface.co/valhalla/distilbart-mnli-12-3)         | 88.1        | 88.19          |
+| [distilbart-mnli-12-6](https://huggingface.co/valhalla/distilbart-mnli-12-6)         | 89.19       | 89.01          |
+| [distilbart-mnli-12-9](https://huggingface.co/valhalla/distilbart-mnli-12-9)         | 89.56       | 89.52          |
+
+
+This is a very simple and effective technique, as we can see the performance drop is very little.
+
+Detailed performace trade-offs will be posted in this [sheet](https://docs.google.com/spreadsheets/d/1dQeUvAKpScLuhDV1afaPJRRAE55s2LpIzDVA5xfqxvk/edit?usp=sharing).
+
+
+## Fine-tuning
+If you want to train these models yourself, clone the [distillbart-mnli repo](https://github.com/patil-suraj/distillbart-mnli) and follow the steps below
+
+Clone and install transformers from source
+```bash
+git clone https://github.com/huggingface/transformers.git
+pip install -qqq -U ./transformers
+```
+
+Download MNLI data
+```bash
+python transformers/utils/download_glue_data.py --data_dir glue_data --tasks MNLI
+```
+
+Create student model
+```bash
+python create_student.py \
+  --teacher_model_name_or_path facebook/bart-large-mnli \
+  --student_encoder_layers 12 \
+  --student_decoder_layers 6 \
+  --save_path student-bart-mnli-12-6 \
+```
+
+Start fine-tuning
+```bash
+python run_glue.py args.json
+```
+
+You can find the logs of these trained models in this [wandb project](https://wandb.ai/psuraj/distilbart-mnli).

model_cards/valhalla/distilbart-mnli-12-3/README.md

@@ -0,0 +1,58 @@
+---
+datasets:
+- mnli
+tags:
+- distilbart
+- distilbart-mnli
+---
+
+# DistilBart-MNLI
+
+distilbart-mnli is the distilled version of bart-large-mnli created using the **No Teacher Distillation** technique proposed for BART summarisation by Huggingface, [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq#distilbart).
+
+We just copy alternating layers from `bart-large-mnli` and finetune more on the same data. 
+
+
+|                                                                                      | matched acc | mismatched acc |
+| ------------------------------------------------------------------------------------ | ----------- | -------------- |
+| [bart-large-mnli](https://huggingface.co/facebook/bart-large-mnli) (baseline, 12-12) | 89.9        | 90.01          |
+| [distilbart-mnli-12-1](https://huggingface.co/valhalla/distilbart-mnli-12-1)         | 87.08       | 87.5           |
+| [distilbart-mnli-12-3](https://huggingface.co/valhalla/distilbart-mnli-12-3)         | 88.1        | 88.19          |
+| [distilbart-mnli-12-6](https://huggingface.co/valhalla/distilbart-mnli-12-6)         | 89.19       | 89.01          |
+| [distilbart-mnli-12-9](https://huggingface.co/valhalla/distilbart-mnli-12-9)         | 89.56       | 89.52          |
+
+
+This is a very simple and effective technique, as we can see the performance drop is very little.
+
+Detailed performace trade-offs will be posted in this [sheet](https://docs.google.com/spreadsheets/d/1dQeUvAKpScLuhDV1afaPJRRAE55s2LpIzDVA5xfqxvk/edit?usp=sharing).
+
+
+## Fine-tuning
+If you want to train these models yourself, clone the [distillbart-mnli repo](https://github.com/patil-suraj/distillbart-mnli) and follow the steps below
+
+Clone and install transformers from source
+```bash
+git clone https://github.com/huggingface/transformers.git
+pip install -qqq -U ./transformers
+```
+
+Download MNLI data
+```bash
+python transformers/utils/download_glue_data.py --data_dir glue_data --tasks MNLI
+```
+
+Create student model
+```bash
+python create_student.py \
+  --teacher_model_name_or_path facebook/bart-large-mnli \
+  --student_encoder_layers 12 \
+  --student_decoder_layers 6 \
+  --save_path student-bart-mnli-12-6 \
+```
+
+Start fine-tuning
+```bash
+python run_glue.py args.json
+```
+
+You can find the logs of these trained models in this [wandb project](https://wandb.ai/psuraj/distilbart-mnli).

model_cards/valhalla/distilbart-mnli-12-6/README.md

@@ -0,0 +1,58 @@
+---
+datasets:
+- mnli
+tags:
+- distilbart
+- distilbart-mnli
+---
+
+# DistilBart-MNLI
+
+distilbart-mnli is the distilled version of bart-large-mnli created using the **No Teacher Distillation** technique proposed for BART summarisation by Huggingface, [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq#distilbart).
+
+We just copy alternating layers from `bart-large-mnli` and finetune more on the same data. 
+
+
+|                                                                                      | matched acc | mismatched acc |
+| ------------------------------------------------------------------------------------ | ----------- | -------------- |
+| [bart-large-mnli](https://huggingface.co/facebook/bart-large-mnli) (baseline, 12-12) | 89.9        | 90.01          |
+| [distilbart-mnli-12-1](https://huggingface.co/valhalla/distilbart-mnli-12-1)         | 87.08       | 87.5           |
+| [distilbart-mnli-12-3](https://huggingface.co/valhalla/distilbart-mnli-12-3)         | 88.1        | 88.19          |
+| [distilbart-mnli-12-6](https://huggingface.co/valhalla/distilbart-mnli-12-6)         | 89.19       | 89.01          |
+| [distilbart-mnli-12-9](https://huggingface.co/valhalla/distilbart-mnli-12-9)         | 89.56       | 89.52          |
+
+
+This is a very simple and effective technique, as we can see the performance drop is very little.
+
+Detailed performace trade-offs will be posted in this [sheet](https://docs.google.com/spreadsheets/d/1dQeUvAKpScLuhDV1afaPJRRAE55s2LpIzDVA5xfqxvk/edit?usp=sharing).
+
+
+## Fine-tuning
+If you want to train these models yourself, clone the [distillbart-mnli repo](https://github.com/patil-suraj/distillbart-mnli) and follow the steps below
+
+Clone and install transformers from source
+```bash
+git clone https://github.com/huggingface/transformers.git
+pip install -qqq -U ./transformers
+```
+
+Download MNLI data
+```bash
+python transformers/utils/download_glue_data.py --data_dir glue_data --tasks MNLI
+```
+
+Create student model
+```bash
+python create_student.py \
+  --teacher_model_name_or_path facebook/bart-large-mnli \
+  --student_encoder_layers 12 \
+  --student_decoder_layers 6 \
+  --save_path student-bart-mnli-12-6 \
+```
+
+Start fine-tuning
+```bash
+python run_glue.py args.json
+```
+
+You can find the logs of these trained models in this [wandb project](https://wandb.ai/psuraj/distilbart-mnli).

model_cards/valhalla/distilbart-mnli-12-9/README.md

@@ -0,0 +1,58 @@
+---
+datasets:
+- mnli
+tags:
+- distilbart
+- distilbart-mnli
+---
+
+# DistilBart-MNLI
+
+distilbart-mnli is the distilled version of bart-large-mnli created using the **No Teacher Distillation** technique proposed for BART summarisation by Huggingface, [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq#distilbart).
+
+We just copy alternating layers from `bart-large-mnli` and finetune more on the same data. 
+
+
+|                                                                                      | matched acc | mismatched acc |
+| ------------------------------------------------------------------------------------ | ----------- | -------------- |
+| [bart-large-mnli](https://huggingface.co/facebook/bart-large-mnli) (baseline, 12-12) | 89.9        | 90.01          |
+| [distilbart-mnli-12-1](https://huggingface.co/valhalla/distilbart-mnli-12-1)         | 87.08       | 87.5           |
+| [distilbart-mnli-12-3](https://huggingface.co/valhalla/distilbart-mnli-12-3)         | 88.1        | 88.19          |
+| [distilbart-mnli-12-6](https://huggingface.co/valhalla/distilbart-mnli-12-6)         | 89.19       | 89.01          |
+| [distilbart-mnli-12-9](https://huggingface.co/valhalla/distilbart-mnli-12-9)         | 89.56       | 89.52          |
+
+
+This is a very simple and effective technique, as we can see the performance drop is very little.
+
+Detailed performace trade-offs will be posted in this [sheet](https://docs.google.com/spreadsheets/d/1dQeUvAKpScLuhDV1afaPJRRAE55s2LpIzDVA5xfqxvk/edit?usp=sharing).
+
+
+## Fine-tuning
+If you want to train these models yourself, clone the [distillbart-mnli repo](https://github.com/patil-suraj/distillbart-mnli) and follow the steps below
+
+Clone and install transformers from source
+```bash
+git clone https://github.com/huggingface/transformers.git
+pip install -qqq -U ./transformers
+```
+
+Download MNLI data
+```bash
+python transformers/utils/download_glue_data.py --data_dir glue_data --tasks MNLI
+```
+
+Create student model
+```bash
+python create_student.py \
+  --teacher_model_name_or_path facebook/bart-large-mnli \
+  --student_encoder_layers 12 \
+  --student_decoder_layers 6 \
+  --save_path student-bart-mnli-12-6 \
+```
+
+Start fine-tuning
+```bash
+python run_glue.py args.json
+```
+
+You can find the logs of these trained models in this [wandb project](https://wandb.ai/psuraj/distilbart-mnli).

model_cards/vinai/bertweet-base/README.md

@@ -0,0 +1,80 @@
+# <a name="introduction"></a> BERTweet: A pre-trained language model for English Tweets 
+
+ - BERTweet is the first public large-scale language model pre-trained for English Tweets. BERTweet is trained based on the [RoBERTa](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md)  pre-training procedure, using the same model configuration as [BERT-base](https://github.com/google-research/bert). 
+ - The corpus used to pre-train BERTweet consists of 850M English Tweets (16B word tokens ~ 80GB), containing 845M Tweets streamed from 01/2012 to 08/2019 and 5M Tweets related to the **COVID-19** pandemic. 
+ - BERTweet does better than its competitors RoBERTa-base and [XLM-R-base](https://arxiv.org/abs/1911.02116) and outperforms previous state-of-the-art models on three downstream Tweet NLP tasks of Part-of-speech tagging, Named entity recognition and text classification.
+
+The general architecture and experimental results of BERTweet can be found in our [paper](https://arxiv.org/abs/2005.10200):
+
+    @inproceedings{bertweet,
+    title     = {{BERTweet: A pre-trained language model for English Tweets}},
+    author    = {Dat Quoc Nguyen and Thanh Vu and Anh Tuan Nguyen},
+    booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
+    year      = {2020}
+    }
+
+**Please CITE** our paper when BERTweet is used to help produce published results or is incorporated into other software.
+
+For further information or requests, please go to [BERTweet's homepage](https://github.com/VinAIResearch/BERTweet)!
+
+### <a name="install2"></a> Installation 
+
+ -  Python 3.6+, and PyTorch 1.1.0+ (or TensorFlow 2.0+)
+ -  Install `transformers`:
+    - `git clone https://github.com/huggingface/transformers.git`
+    - `cd transformers`
+    - `pip3 install --upgrade .`
+ - Install `emoji`: `pip3 install emoji`
+
+### <a name="models2"></a> Pre-trained models 
+
+
+Model | #params | Arch. | Pre-training data
+---|---|---|---
+`vinai/bertweet-base` | 135M | base | 845M English Tweets (cased)
+`vinai/bertweet-covid19-base-cased` | 135M | base | 23M COVID-19 English Tweets (cased)
+`vinai/bertweet-covid19-base-uncased` | 135M | base | 23M COVID-19 English Tweets (uncased)
+
+Two pre-trained models `vinai/bertweet-covid19-base-cased` and `vinai/bertweet-covid19-base-uncased` are resulted by further pre-training the pre-trained model `vinai/bertweet-base` on a  corpus of 23M COVID-19 English Tweets for 40 epochs.  
+
+### <a name="usage2"></a> Example usage 
+
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer 
+
+bertweet = AutoModel.from_pretrained("vinai/bertweet-base")
+tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base")
+
+# INPUT TWEET IS ALREADY NORMALIZED!
+line = "SC has first two presumptive cases of coronavirus , DHEC confirms HTTPURL via @USER :cry:"
+
+input_ids = torch.tensor([tokenizer.encode(line)])
+
+with torch.no_grad():
+    features = bertweet(input_ids)  # Models outputs are now tuples
+    
+## With TensorFlow 2.0+:
+# from transformers import TFAutoModel
+# bertweet = TFAutoModel.from_pretrained("vinai/bertweet-base")
+```
+
+### <a name="preprocess"></a> Normalize raw input Tweets 
+
+Before applying `fastBPE` to the pre-training corpus of 850M English Tweets, we tokenized these  Tweets using `TweetTokenizer` from the NLTK toolkit and used the `emoji` package to translate emotion icons into text strings (here, each icon is referred to as a word token).   We also normalized the Tweets by converting user mentions and web/url links into special tokens `@USER` and `HTTPURL`, respectively. Thus it is recommended to also apply the same pre-processing step for BERTweet-based downstream applications w.r.t. the raw input Tweets. BERTweet provides this pre-processing step by enabling the `normalization` argument. 
+
+```python
+import torch
+from transformers import AutoTokenizer
+
+# Load the AutoTokenizer with a normalization mode if the input Tweet is raw
+tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base", normalization=True)
+
+# from transformers import BertweetTokenizer
+# tokenizer = BertweetTokenizer.from_pretrained("vinai/bertweet-base", normalization=True)
+
+line = "SC has first two presumptive cases of coronavirus, DHEC confirms https://postandcourier.com/health/covid19/sc-has-first-two-presumptive-cases-of-coronavirus-dhec-confirms/article_bddfe4ae-5fd3-11ea-9ce4-5f495366cee6.html?utm_medium=social&utm_source=twitter&utm_campaign=user-share… via @postandcourier"
+
+input_ids = torch.tensor([tokenizer.encode(line)])
+```

model_cards/vinai/bertweet-covid19-base-cased/README.md

@@ -0,0 +1,80 @@
+# <a name="introduction"></a> BERTweet: A pre-trained language model for English Tweets 
+
+ - BERTweet is the first public large-scale language model pre-trained for English Tweets. BERTweet is trained based on the [RoBERTa](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md)  pre-training procedure, using the same model configuration as [BERT-base](https://github.com/google-research/bert). 
+ - The corpus used to pre-train BERTweet consists of 850M English Tweets (16B word tokens ~ 80GB), containing 845M Tweets streamed from 01/2012 to 08/2019 and 5M Tweets related to the **COVID-19** pandemic. 
+ - BERTweet does better than its competitors RoBERTa-base and [XLM-R-base](https://arxiv.org/abs/1911.02116) and outperforms previous state-of-the-art models on three downstream Tweet NLP tasks of Part-of-speech tagging, Named entity recognition and text classification.
+
+The general architecture and experimental results of BERTweet can be found in our [paper](https://arxiv.org/abs/2005.10200):
+
+    @inproceedings{bertweet,
+    title     = {{BERTweet: A pre-trained language model for English Tweets}},
+    author    = {Dat Quoc Nguyen and Thanh Vu and Anh Tuan Nguyen},
+    booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
+    year      = {2020}
+    }
+
+**Please CITE** our paper when BERTweet is used to help produce published results or is incorporated into other software.
+
+For further information or requests, please go to [BERTweet's homepage](https://github.com/VinAIResearch/BERTweet)!
+
+### <a name="install2"></a> Installation 
+
+ -  Python 3.6+, and PyTorch 1.1.0+ (or TensorFlow 2.0+)
+ -  Install `transformers`:
+    - `git clone https://github.com/huggingface/transformers.git`
+    - `cd transformers`
+    - `pip3 install --upgrade .`
+ - Install `emoji`: `pip3 install emoji`
+
+### <a name="models2"></a> Pre-trained models 
+
+
+Model | #params | Arch. | Pre-training data
+---|---|---|---
+`vinai/bertweet-base` | 135M | base | 845M English Tweets (cased)
+`vinai/bertweet-covid19-base-cased` | 135M | base | 23M COVID-19 English Tweets (cased)
+`vinai/bertweet-covid19-base-uncased` | 135M | base | 23M COVID-19 English Tweets (uncased)
+
+Two pre-trained models `vinai/bertweet-covid19-base-cased` and `vinai/bertweet-covid19-base-uncased` are resulted by further pre-training the pre-trained model `vinai/bertweet-base` on a  corpus of 23M COVID-19 English Tweets for 40 epochs.  
+
+### <a name="usage2"></a> Example usage 
+
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer 
+
+bertweet = AutoModel.from_pretrained("vinai/bertweet-covid19-base-cased")
+tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-covid19-base-cased")
+
+# INPUT TWEET IS ALREADY NORMALIZED!
+line = "SC has first two presumptive cases of coronavirus , DHEC confirms HTTPURL via @USER :cry:"
+
+input_ids = torch.tensor([tokenizer.encode(line)])
+
+with torch.no_grad():
+    features = bertweet(input_ids)  # Models outputs are now tuples
+    
+## With TensorFlow 2.0+:
+# from transformers import TFAutoModel
+# bertweet = TFAutoModel.from_pretrained("vinai/bertweet-covid19-base-cased")
+```
+
+### <a name="preprocess"></a> Normalize raw input Tweets 
+
+Before applying `fastBPE` to the pre-training corpus of 850M English Tweets, we tokenized these  Tweets using `TweetTokenizer` from the NLTK toolkit and used the `emoji` package to translate emotion icons into text strings (here, each icon is referred to as a word token).   We also normalized the Tweets by converting user mentions and web/url links into special tokens `@USER` and `HTTPURL`, respectively. Thus it is recommended to also apply the same pre-processing step for BERTweet-based downstream applications w.r.t. the raw input Tweets. BERTweet provides this pre-processing step by enabling the `normalization` argument. 
+
+```python
+import torch
+from transformers import AutoTokenizer
+
+# Load the AutoTokenizer with a normalization mode if the input Tweet is raw
+tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-covid19-base-cased", normalization=True)
+
+# from transformers import BertweetTokenizer
+# tokenizer = BertweetTokenizer.from_pretrained("vinai/bertweet-covid19-base-cased", normalization=True)
+
+line = "SC has first two presumptive cases of coronavirus, DHEC confirms https://postandcourier.com/health/covid19/sc-has-first-two-presumptive-cases-of-coronavirus-dhec-confirms/article_bddfe4ae-5fd3-11ea-9ce4-5f495366cee6.html?utm_medium=social&utm_source=twitter&utm_campaign=user-share… via @postandcourier"
+
+input_ids = torch.tensor([tokenizer.encode(line)])
+```

model_cards/vinai/bertweet-covid19-base-uncased/README.md

@@ -0,0 +1,80 @@
+# <a name="introduction"></a> BERTweet: A pre-trained language model for English Tweets 
+
+ - BERTweet is the first public large-scale language model pre-trained for English Tweets. BERTweet is trained based on the [RoBERTa](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md)  pre-training procedure, using the same model configuration as [BERT-base](https://github.com/google-research/bert). 
+ - The corpus used to pre-train BERTweet consists of 850M English Tweets (16B word tokens ~ 80GB), containing 845M Tweets streamed from 01/2012 to 08/2019 and 5M Tweets related to the **COVID-19** pandemic. 
+ - BERTweet does better than its competitors RoBERTa-base and [XLM-R-base](https://arxiv.org/abs/1911.02116) and outperforms previous state-of-the-art models on three downstream Tweet NLP tasks of Part-of-speech tagging, Named entity recognition and text classification.
+
+The general architecture and experimental results of BERTweet can be found in our [paper](https://arxiv.org/abs/2005.10200):
+
+    @inproceedings{bertweet,
+    title     = {{BERTweet: A pre-trained language model for English Tweets}},
+    author    = {Dat Quoc Nguyen and Thanh Vu and Anh Tuan Nguyen},
+    booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
+    year      = {2020}
+    }
+
+**Please CITE** our paper when BERTweet is used to help produce published results or is incorporated into other software.
+
+For further information or requests, please go to [BERTweet's homepage](https://github.com/VinAIResearch/BERTweet)!
+
+### <a name="install2"></a> Installation 
+
+ -  Python 3.6+, and PyTorch 1.1.0+ (or TensorFlow 2.0+)
+ -  Install `transformers`:
+    - `git clone https://github.com/huggingface/transformers.git`
+    - `cd transformers`
+    - `pip3 install --upgrade .`
+ - Install `emoji`: `pip3 install emoji`
+
+### <a name="models2"></a> Pre-trained models 
+
+
+Model | #params | Arch. | Pre-training data
+---|---|---|---
+`vinai/bertweet-base` | 135M | base | 845M English Tweets (cased)
+`vinai/bertweet-covid19-base-cased` | 135M | base | 23M COVID-19 English Tweets (cased)
+`vinai/bertweet-covid19-base-uncased` | 135M | base | 23M COVID-19 English Tweets (uncased)
+
+Two pre-trained models `vinai/bertweet-covid19-base-cased` and `vinai/bertweet-covid19-base-uncased` are resulted by further pre-training the pre-trained model `vinai/bertweet-base` on a  corpus of 23M COVID-19 English Tweets for 40 epochs.  
+
+### <a name="usage2"></a> Example usage 
+
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer 
+
+bertweet = AutoModel.from_pretrained("vinai/bertweet-covid19-base-uncased")
+tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-covid19-base-uncased")
+
+# INPUT TWEET IS ALREADY NORMALIZED!
+line = "SC has first two presumptive cases of coronavirus , DHEC confirms HTTPURL via @USER :cry:"
+
+input_ids = torch.tensor([tokenizer.encode(line)])
+
+with torch.no_grad():
+    features = bertweet(input_ids)  # Models outputs are now tuples
+    
+## With TensorFlow 2.0+:
+# from transformers import TFAutoModel
+# bertweet = TFAutoModel.from_pretrained("vinai/bertweet-covid19-base-uncased")
+```
+
+### <a name="preprocess"></a> Normalize raw input Tweets 
+
+Before applying `fastBPE` to the pre-training corpus of 850M English Tweets, we tokenized these  Tweets using `TweetTokenizer` from the NLTK toolkit and used the `emoji` package to translate emotion icons into text strings (here, each icon is referred to as a word token).   We also normalized the Tweets by converting user mentions and web/url links into special tokens `@USER` and `HTTPURL`, respectively. Thus it is recommended to also apply the same pre-processing step for BERTweet-based downstream applications w.r.t. the raw input Tweets. BERTweet provides this pre-processing step by enabling the `normalization` argument. 
+
+```python
+import torch
+from transformers import AutoTokenizer
+
+# Load the AutoTokenizer with a normalization mode if the input Tweet is raw
+tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-covid19-base-uncased", normalization=True)
+
+# from transformers import BertweetTokenizer
+# tokenizer = BertweetTokenizer.from_pretrained("vinai/bertweet-covid19-base-uncased", normalization=True)
+
+line = "SC has first two presumptive cases of coronavirus, DHEC confirms https://postandcourier.com/health/covid19/sc-has-first-two-presumptive-cases-of-coronavirus-dhec-confirms/article_bddfe4ae-5fd3-11ea-9ce4-5f495366cee6.html?utm_medium=social&utm_source=twitter&utm_campaign=user-share… via @postandcourier"
+
+input_ids = torch.tensor([tokenizer.encode(line)])
+```

model_cards/vinai/phobert-base/README.md

@@ -0,0 +1,55 @@
+# <a name="introduction"></a> PhoBERT: Pre-trained language models for Vietnamese 
+  
+Pre-trained PhoBERT models are the state-of-the-art language models for Vietnamese ([Pho](https://en.wikipedia.org/wiki/Pho), i.e. "Phở", is a popular food in Vietnam):
+
+ - Two PhoBERT versions of "base" and "large" are the first public large-scale monolingual language models pre-trained for Vietnamese. PhoBERT pre-training approach is based on [RoBERTa](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md)  which optimizes the [BERT](https://github.com/google-research/bert) pre-training procedure for more robust performance.
+ - PhoBERT outperforms previous monolingual and multilingual approaches, obtaining new state-of-the-art performances on four downstream Vietnamese NLP tasks of Part-of-speech tagging, Dependency parsing, Named-entity recognition and Natural language inference.
+
+The general architecture and experimental results of PhoBERT can be found in our EMNLP-2020 Findings [paper](https://arxiv.org/abs/2003.00744):
+
+    @article{phobert,
+    title     = {{PhoBERT: Pre-trained language models for Vietnamese}},
+    author    = {Dat Quoc Nguyen and Anh Tuan Nguyen},
+    journal   = {Findings of EMNLP},
+    year      = {2020}
+    }
+
+**Please CITE** our paper when PhoBERT is used to help produce published results or is incorporated into other software.
+
+For further information or requests, please go to [PhoBERT's homepage](https://github.com/VinAIResearch/PhoBERT)!
+
+### Installation <a name="install2"></a>
+ -  Python 3.6+, and PyTorch 1.1.0+ (or TensorFlow 2.0+)
+ -  Install `transformers`:
+        - `git clone https://github.com/huggingface/transformers.git`
+        - `cd transformers`
+        - `pip3 install --upgrade .`
+
+### Pre-trained models <a name="models2"></a>
+
+Model | #params | Arch.  | Pre-training data
+---|---|---|---
+`vinai/phobert-base` | 135M | base | 20GB  of texts
+`vinai/phobert-large` | 370M | large | 20GB  of texts
+
+### Example usage <a name="usage2"></a>
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+phobert = AutoModel.from_pretrained("vinai/phobert-base")
+tokenizer = AutoTokenizer.from_pretrained("vinai/phobert-base")
+
+# INPUT TEXT MUST BE ALREADY WORD-SEGMENTED!
+line = "Tôi là sinh_viên trường đại_học Công_nghệ ."
+
+input_ids = torch.tensor([tokenizer.encode(line)])
+
+with torch.no_grad():
+    features = phobert(input_ids)  # Models outputs are now tuples
+
+## With TensorFlow 2.0+:
+# from transformers import TFAutoModel
+# phobert = TFAutoModel.from_pretrained("vinai/phobert-base")
+```

model_cards/vinai/phobert-large/README.md

@@ -0,0 +1,55 @@
+# <a name="introduction"></a> PhoBERT: Pre-trained language models for Vietnamese 
+  
+Pre-trained PhoBERT models are the state-of-the-art language models for Vietnamese ([Pho](https://en.wikipedia.org/wiki/Pho), i.e. "Phở", is a popular food in Vietnam):
+
+ - Two PhoBERT versions of "base" and "large" are the first public large-scale monolingual language models pre-trained for Vietnamese. PhoBERT pre-training approach is based on [RoBERTa](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md)  which optimizes the [BERT](https://github.com/google-research/bert) pre-training procedure for more robust performance.
+ - PhoBERT outperforms previous monolingual and multilingual approaches, obtaining new state-of-the-art performances on four downstream Vietnamese NLP tasks of Part-of-speech tagging, Dependency parsing, Named-entity recognition and Natural language inference.
+
+The general architecture and experimental results of PhoBERT can be found in our EMNLP-2020 Findings [paper](https://arxiv.org/abs/2003.00744):
+
+    @article{phobert,
+    title     = {{PhoBERT: Pre-trained language models for Vietnamese}},
+    author    = {Dat Quoc Nguyen and Anh Tuan Nguyen},
+    journal   = {Findings of EMNLP},
+    year      = {2020}
+    }
+
+**Please CITE** our paper when PhoBERT is used to help produce published results or is incorporated into other software.
+
+For further information or requests, please go to [PhoBERT's homepage](https://github.com/VinAIResearch/PhoBERT)!
+
+### Installation <a name="install2"></a>
+ -  Python 3.6+, and PyTorch 1.1.0+ (or TensorFlow 2.0+)
+ -  Install `transformers`:
+        - `git clone https://github.com/huggingface/transformers.git`
+        - `cd transformers`
+        - `pip3 install --upgrade .`
+
+### Pre-trained models <a name="models2"></a>
+
+Model | #params | Arch.  | Pre-training data
+---|---|---|---
+`vinai/phobert-base` | 135M | base | 20GB  of texts
+`vinai/phobert-large` | 370M | large | 20GB  of texts
+
+### Example usage <a name="usage2"></a>
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+phobert = AutoModel.from_pretrained("vinai/phobert-large")
+tokenizer = AutoTokenizer.from_pretrained("vinai/phobert-large")
+
+# INPUT TEXT MUST BE ALREADY WORD-SEGMENTED!
+line = "Tôi là sinh_viên trường đại_học Công_nghệ ."
+
+input_ids = torch.tensor([tokenizer.encode(line)])
+
+with torch.no_grad():
+    features = phobert(input_ids)  # Models outputs are now tuples
+
+## With TensorFlow 2.0+:
+# from transformers import TFAutoModel
+# phobert = TFAutoModel.from_pretrained("vinai/phobert-large")
+```

notebooks/README.md

@@ -44,4 +44,5 @@ Pull Request so it can be included under the Community notebooks.
 |[Expand and Fine Tune Sci-BERT](https://github.com/lordtt13/word-embeddings/blob/master/COVID-19%20Research%20Data/COVID-SciBERT.ipynb)| How to increase vocabulary of a pretrained SciBERT model from AllenAI on the CORD dataset and pipeline it. | [Tanmay Thakur](https://github.com/lordtt13) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1rqAR40goxbAfez1xvF3hBJphSCsvXmh8)|
 |[Fine-tune Electra and interpret with Integrated Gradients](https://github.com/elsanns/xai-nlp-notebooks/blob/master/electra_fine_tune_interpret_captum_ig.ipynb) | How to fine-tune Electra for sentiment analysis and interpret predictions with Captum Integrated Gradients | [Eliza Szczechla](https://elsanns.github.io) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/elsanns/xai-nlp-notebooks/blob/master/electra_fine_tune_interpret_captum_ig.ipynb)|
 |[fine-tune a non-English GPT-2 Model with Trainer class](https://github.com/philschmid/fine-tune-GPT-2/blob/master/Fine_tune_a_non_English_GPT_2_Model_with_Huggingface.ipynb) | How to fine-tune a non-English GPT-2 Model with Trainer class | [Philipp Schmid](https://www.philschmid.de) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/philschmid/fine-tune-GPT-2/blob/master/Fine_tune_a_non_English_GPT_2_Model_with_Huggingface.ipynb)|
-|[Fine-tune a DistilBERT Model for Multi Label Classification task](https://github.com/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb) | How to fine-tune a DistilBERT Model for Multi Label Classification task | [Dhaval Taunk](https://github.com/DhavalTaunk08) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb)|
+|[Fine-tune a DistilBERT Model for Multi Label Classification task](https://github.com/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb) | How to fine-tune a DistilBERT Model for Multi Label Classification task | [Dhaval Taunk](https://github.com/DhavalTaunk08) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb)|
+|[Fine-tune ALBERT for sentence-pair classification](https://github.com/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb) | How to fine-tune an ALBERT model or another BERT-based model for the sentence-pair classification task | [Nadir El Manouzi](https://github.com/NadirEM) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb)|

scripts/fsmt/fsmt-make-tiny-model.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+# coding: utf-8
+
+# this script creates a tiny model that is useful inside tests, when we just want to test that the machinery works,
+# without needing to the check the quality of the outcomes.
+# it will be used then as "stas/tiny-wmt19-en-de"
+
+from transformers import FSMTTokenizer, FSMTConfig, FSMTForConditionalGeneration
+mname = "facebook/wmt19-en-de"
+tokenizer = FSMTTokenizer.from_pretrained(mname)
+# get the correct vocab sizes, etc. from the master model
+config = FSMTConfig.from_pretrained(mname)
+config.update(dict(
+    d_model=4,
+    encoder_layers=1, decoder_layers=1,
+    encoder_ffn_dim=4, decoder_ffn_dim=4,
+    encoder_attention_heads=1, decoder_attention_heads=1))
+
+tiny_model = FSMTForConditionalGeneration(config)
+print(f"num of params {tiny_model.num_parameters()}")
+# Test it
+batch = tokenizer.prepare_seq2seq_batch(["Making tiny model"])
+outputs = tiny_model(**batch, return_dict=True)
+
+print(len(outputs.logits[0]))
+# Save
+mname_tiny = "tiny-wmt19-en-de"
+tiny_model.half() # makes it smaller
+tiny_model.save_pretrained(mname_tiny)
+tokenizer.save_pretrained(mname_tiny)
+
+# Upload
+# transformers-cli upload tiny-wmt19-en-de

scripts/fsmt/gen-card-allenai-wmt16.py

@@ -25,19 +25,19 @@ def write_model_card(model_card_dir, src_lang, tgt_lang, model_name):
 
     readme = f"""
 ---
-
-language: {src_lang}, {tgt_lang}
+language:
+- {src_lang}
+- {tgt_lang}
 thumbnail:
 tags:
 - translation
 - wmt16
 - allenai
-license: Apache 2.0
+license: apache-2.0
 datasets:
-- http://www.statmt.org/wmt16/ ([test-set](http://matrix.statmt.org/test_sets/newstest2016.tgz?1504722372))
-
+- wmt16
 metrics:
-- http://www.statmt.org/wmt16/metrics-task.html
+- bleu
 ---
 
 # FSMT
@@ -54,16 +54,6 @@ All 3 models are available:
 * [wmt16-en-de-dist-6-1](https://huggingface.co/allenai/wmt16-en-de-dist-6-1)
 * [wmt16-en-de-12-1](https://huggingface.co/allenai/wmt16-en-de-12-1)
 
-```
-@misc{{kasai2020deep,
-    title={{Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation}},
-    author={{Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith}},
-    year={{2020}},
-    eprint={{2006.10369}},
-    archivePrefix={{arXiv}},
-    primaryClass={{cs.CL}}
-}}
-```
 
 ## Intended uses & limitations
 
@@ -118,6 +108,25 @@ echo $PAIR
 PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/{model_name} $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
 ```
 
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt16/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2016.tgz?1504722372)
+
+
+### BibTeX entry and citation info
+
+```
+@misc{{kasai2020deep,
+    title={{Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation}},
+    author={{Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith}},
+    year={{2020}},
+    eprint={{2006.10369}},
+    archivePrefix={{arXiv}},
+    primaryClass={{cs.CL}}
+}}
+```
+
 """
     model_card_dir.mkdir(parents=True, exist_ok=True)
     path = os.path.join(model_card_dir, "README.md")

scripts/fsmt/gen-card-allenai-wmt19.py

@@ -25,30 +25,35 @@ def write_model_card(model_card_dir, src_lang, tgt_lang, model_name):
     readme = f"""
 ---
 
-language: {src_lang}, {tgt_lang}
+language:
+- {src_lang}
+- {tgt_lang}
 thumbnail:
 tags:
 - translation
 - wmt19
 - allenai
-license: Apache 2.0
+license: apache-2.0
 datasets:
-- http://www.statmt.org/wmt19/ ([test-set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561))
+- wmt19
 metrics:
-- http://www.statmt.org/wmt19/metrics-task.html
+- bleu
 ---
 
 # FSMT
 
 ## Model description
 
-This is a ported version of fairseq-based wmt19 transformer created by [jungokasai]](https://github.com/jungokasai/) @ allenai for {src_lang}-{tgt_lang}.
+This is a ported version of fairseq-based [wmt19 transformer](https://github.com/jungokasai/deep-shallow/) for {src_lang}-{tgt_lang}.
+
+For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369).
 
 2 models are available:
 
 * [wmt19-de-en-6-6-big](https://huggingface.co/allenai/wmt19-de-en-6-6-big)
 * [wmt19-de-en-6-6-base](https://huggingface.co/allenai/wmt19-de-en-6-6-base)
 
+
 ## Intended uses & limitations
 
 #### How to use
@@ -73,7 +78,7 @@ print(decoded) # {texts[tgt_lang]}
 
 ## Training data
 
-Pretrained weights were left identical to the original model released by the researcher.
+Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369).
 
 ## Eval results
 
@@ -100,6 +105,25 @@ echo $PAIR
 PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/{model_name} $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
 ```
 
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt19/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)
+
+
+### BibTeX entry and citation info
+
+```
+@misc{{kasai2020deep,
+    title={{Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation}},
+    author={{Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith}},
+    year={{2020}},
+    eprint={{2006.10369}},
+    archivePrefix={{arXiv}},
+    primaryClass={{cs.CL}}
+}}
+```
+
 """
     model_card_dir.mkdir(parents=True, exist_ok=True)
     path = os.path.join(model_card_dir, "README.md")

scripts/fsmt/gen-card-facebook-wmt19.py

@@ -26,19 +26,19 @@ def write_model_card(model_card_dir, src_lang, tgt_lang):
 
     readme = f"""
 ---
-
-<!-- This file has been auto-generated by src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py - DO NOT EDIT or your changes will be lost -->
-
-language: {src_lang}, {tgt_lang}
+language: 
+- {src_lang}
+- {tgt_lang}
 thumbnail:
 tags:
 - translation
 - wmt19
-license: Apache 2.0
+- facebook
+license: apache-2.0
 datasets:
-- http://www.statmt.org/wmt19/ ([test-set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561))
+- wmt19
 metrics:
-- http://www.statmt.org/wmt19/metrics-task.html
+- bleu
 ---
 
 # FSMT
@@ -69,7 +69,7 @@ mname = "facebook/wmt19-{src_lang}-{tgt_lang}"
 tokenizer = FSMTTokenizer.from_pretrained(mname)
 model = FSMTForConditionalGeneration.from_pretrained(mname)
 
-input = "{texts[src_lang]}
+input = "{texts[src_lang]}"
 input_ids = tokenizer.encode(input, return_tensors="pt")
 outputs = model.generate(input_ids)
 decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
@@ -113,6 +113,23 @@ PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py facebook/w
 ```
 note: fairseq reports using a beam of 50, so you should get a slightly higher score if re-run with `--num_beams 50`.
 
+## Data Sources
+
+- [training, etc.](http://www.statmt.org/wmt19/)
+- [test set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@inproceedings{{...,
+  year={{2020}},
+  title={{Facebook FAIR's WMT19 News Translation Task Submission}},
+  author={{Ng, Nathan and Yee, Kyra and Baevski, Alexei and Ott, Myle and Auli, Michael and Edunov, Sergey}},
+  booktitle={{Proc. of WMT}},
+}}
+```
+
 
 ## TODO
 

src/transformers/__init__.py

@@ -30,6 +30,7 @@ from .configuration_dpr import DPR_PRETRAINED_CONFIG_ARCHIVE_MAP, DPRConfig
 from .configuration_electra import ELECTRA_PRETRAINED_CONFIG_ARCHIVE_MAP, ElectraConfig
 from .configuration_encoder_decoder import EncoderDecoderConfig
 from .configuration_flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig
+from .configuration_fsmt import FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP, FSMTConfig
 from .configuration_funnel import FUNNEL_PRETRAINED_CONFIG_ARCHIVE_MAP, FunnelConfig
 from .configuration_gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config
 from .configuration_longformer import LONGFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP, LongformerConfig
@@ -145,6 +146,7 @@ from .tokenization_bart import BartTokenizer, BartTokenizerFast
 from .tokenization_bert import BasicTokenizer, BertTokenizer, BertTokenizerFast, WordpieceTokenizer
 from .tokenization_bert_generation import BertGenerationTokenizer
 from .tokenization_bert_japanese import BertJapaneseTokenizer, CharacterTokenizer, MecabTokenizer
+from .tokenization_bertweet import BertweetTokenizer
 from .tokenization_camembert import CamembertTokenizer
 from .tokenization_ctrl import CTRLTokenizer
 from .tokenization_distilbert import DistilBertTokenizer, DistilBertTokenizerFast
@@ -158,6 +160,7 @@ from .tokenization_dpr import (
 )
 from .tokenization_electra import ElectraTokenizer, ElectraTokenizerFast
 from .tokenization_flaubert import FlaubertTokenizer
+from .tokenization_fsmt import FSMTTokenizer
 from .tokenization_funnel import FunnelTokenizer, FunnelTokenizerFast
 from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast
 from .tokenization_longformer import LongformerTokenizer, LongformerTokenizerFast
@@ -166,6 +169,7 @@ from .tokenization_mbart import MBartTokenizer
 from .tokenization_mobilebert import MobileBertTokenizer, MobileBertTokenizerFast
 from .tokenization_openai import OpenAIGPTTokenizer, OpenAIGPTTokenizerFast
 from .tokenization_pegasus import PegasusTokenizer
+from .tokenization_phobert import PhobertTokenizer
 from .tokenization_reformer import ReformerTokenizer
 from .tokenization_retribert import RetriBertTokenizer, RetriBertTokenizerFast
 from .tokenization_roberta import RobertaTokenizer, RobertaTokenizerFast
@@ -338,6 +342,7 @@ if is_torch_available():
         FlaubertModel,
         FlaubertWithLMHeadModel,
     )
+    from .modeling_fsmt import FSMTForConditionalGeneration, FSMTModel, PretrainedFSMTModel
     from .modeling_funnel import (
         FUNNEL_PRETRAINED_MODEL_ARCHIVE_LIST,
         FunnelBaseModel,

src/transformers/configuration_auto.py

@@ -27,6 +27,7 @@ from .configuration_distilbert import DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
 from .configuration_electra import ELECTRA_PRETRAINED_CONFIG_ARCHIVE_MAP, ElectraConfig
 from .configuration_encoder_decoder import EncoderDecoderConfig
 from .configuration_flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig
+from .configuration_fsmt import FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP, FSMTConfig
 from .configuration_funnel import FUNNEL_PRETRAINED_CONFIG_ARCHIVE_MAP, FunnelConfig
 from .configuration_gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config
 from .configuration_longformer import LONGFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP, LongformerConfig
@@ -66,6 +67,7 @@ ALL_PRETRAINED_CONFIG_ARCHIVE_MAP = dict(
         T5_PRETRAINED_CONFIG_ARCHIVE_MAP,
         XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
         FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
+        FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP,
         ELECTRA_PRETRAINED_CONFIG_ARCHIVE_MAP,
         LONGFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP,
         RETRIBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
@@ -94,6 +96,7 @@ CONFIG_MAPPING = OrderedDict(
         ("longformer", LongformerConfig),
         ("roberta", RobertaConfig),
         ("flaubert", FlaubertConfig),
+        ("fsmt", FSMTConfig),
         ("bert", BertConfig),
         ("openai-gpt", OpenAIGPTConfig),
         ("gpt2", GPT2Config),
@@ -126,6 +129,7 @@ MODEL_NAMES_MAPPING = OrderedDict(
         ("longformer", "Longformer"),
         ("roberta", "RoBERTa"),
         ("flaubert", "FlauBERT"),
+        ("fsmt", "FairSeq Machine-Translation"),
         ("bert", "BERT"),
         ("openai-gpt", "OpenAI GPT"),
         ("gpt2", "OpenAI GPT-2"),

src/transformers/configuration_fsmt.py

@@ -0,0 +1,223 @@
+# coding=utf-8
+# Copyright 2019-present, Facebook, Inc and the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" FSMT configuration """
+
+
+import copy
+
+from .configuration_utils import PretrainedConfig
+from .file_utils import add_start_docstrings_to_callable
+from .utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
+
+
+FSMT_CONFIG_ARGS_DOC = r"""
+    Args:
+        langs (:obj:`List[str]`):
+            source language, target_language (e.g. ['en', 'ru'])
+        src_vocab_size (:obj:`int`):
+            defines the different tokens that can be represented by `inputs_ids` passed to the forward
+            method in the encoder.
+        tgt_vocab_size (:obj:`int`):
+            defines the different tokens that can be represented by `inputs_ids` passed to the forward
+            method in the decoder.
+        d_model (:obj:`int`, `optional`, defaults to 1024):
+            Dimensionality of the layers and the pooler layer.
+        encoder_layers (:obj:`int`, `optional`, defaults to 12):
+            Number of encoder layers, 16 for pegasus, 6 for bart-base and marian
+        decoder_layers (:obj:`int`, `optional`, defaults to 12):
+            Number of decoder layers, 16 for pegasus, 6 for bart-base and marian
+        encoder_attention_heads (:obj:`int`, `optional`, defaults to 16):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        decoder_attention_heads (:obj:`int`, `optional`, defaults to 16):
+            Number of attention heads for each attention layer in the Transformer decoder.
+        decoder_ffn_dim (:obj:`int`, `optional`, defaults to 4096):
+            Dimensionality of the "intermediate" (i.e., feed-forward) layer in decoder.
+        encoder_ffn_dim (:obj:`int`, `optional`, defaults to 4096):
+            Dimensionality of the "intermediate" (i.e., feed-forward) layer in decoder.
+        activation_function (:obj:`str` or :obj:`function`, `optional`, defaults to "relu"):
+            The non-linear activation function (function or string) in the encoder and pooler.
+            If string, "gelu", "relu", "swish" and "gelu_new" are supported.
+        dropout (:obj:`float`, `optional`, defaults to 0.1):
+            The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
+        attention_dropout (:obj:`float`, `optional`, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+        activation_dropout (:obj:`float`, `optional`, defaults to 0.0):
+            The dropout ratio for activations inside the fully connected layer.
+        max_position_embeddings (:obj:`int`, `optional`, defaults to 1024):
+            The maximum sequence length that this model might ever be used with.
+            Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
+        init_std (:obj:`float`, `optional`, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        scale_embedding (:obj:`bool`, `optional`, defaults to :obj:`True`):
+            Scale embeddings by diving by sqrt(d_model).
+        bos_token_id (:obj:`int`, `optional`, defaults to 0)
+            Beginning of stream token id.
+        pad_token_id (:obj:`int`, `optional`, defaults to 1)
+            Padding token id.
+        eos_token_id (:obj:`int`, `optional`, defaults to 2)
+            End of stream token id.
+        decoder_start_token_id (:obj:`int`, `optional`):
+            This model starts decoding with `eos_token_id`
+        encoder_layerdrop: (:obj:`float`, `optional`, defaults to 0.0):
+            Google "layerdrop arxiv", as its not explainable in one line.
+        decoder_layerdrop: (:obj:`float`, `optional`, defaults to 0.0):
+            Google "layerdrop arxiv", as its not explainable in one line.
+        is_encoder_decoder (:obj:`bool`, `optional`, defaults to :obj:`True`):
+            Whether this is an encoder/decoder model.
+        tie_word_embeddings (:obj:`bool`, `optional`, defaults to :obj:`False`):
+            Whether to tie input and output embeddings.
+        num_beams (:obj:`int`, `optional`, defaults to 5)
+            Number of beams for beam search that will be used by default in the :obj:`generate` method
+            of the model. 1 means no beam search.
+        length_penalty (:obj:`float`, `optional`, defaults to 1)
+            Exponential penalty to the length that will be used by default in the :obj:`generate` method
+            of the model.
+        early_stopping (:obj:`bool`, `optional`, defaults to :obj:`False`)
+            Flag that will be used by default in the :obj:`generate` method of the model. Whether to stop
+            the beam search when at least ``num_beams`` sentences are finished per batch or not.
+"""
+
+
+class DecoderConfig(PretrainedConfig):
+    r"""
+    Configuration class for FSMT's decoder specific things.
+    note: this is a private helper class
+    """
+    model_type = "fsmt_decoder"
+
+    def __init__(self, vocab_size=0, bos_token_id=0):
+        super().__init__()
+        self.vocab_size = vocab_size
+        self.bos_token_id = bos_token_id
+
+
+@add_start_docstrings_to_callable(FSMT_CONFIG_ARGS_DOC)
+class FSMTConfig(PretrainedConfig):
+    r"""
+    Configuration class for FSMT.
+    """
+    model_type = "fsmt"
+
+    # update the defaults from config file
+    def __init__(
+        self,
+        langs,
+        src_vocab_size,
+        tgt_vocab_size,
+        activation_function="relu",
+        d_model=1024,
+        max_length=200,
+        max_position_embeddings=1024,
+        encoder_ffn_dim=4096,
+        encoder_layers=12,
+        encoder_attention_heads=16,
+        encoder_layerdrop=0.0,
+        decoder_ffn_dim=4096,
+        decoder_layers=12,
+        decoder_attention_heads=16,
+        decoder_layerdrop=0.0,
+        attention_dropout=0.0,
+        dropout=0.1,
+        activation_dropout=0.0,
+        init_std=0.02,
+        pad_token_id=1,
+        bos_token_id=0,
+        eos_token_id=2,
+        decoder_start_token_id=2,
+        is_encoder_decoder=True,
+        scale_embedding=True,
+        tie_word_embeddings=False,
+        num_beams=5,
+        length_penalty=1.0,
+        early_stopping=False,
+        **common_kwargs
+    ):
+        r"""
+        :class:`~transformers.FSMTConfig` is the configuration class for `FSMTModel`.
+
+        Examples::
+
+            >>> from transformers import FSMTConfig, FSMTModel
+
+            >>> config = FSMTConfig.from_pretrained('facebook/wmt19-en-ru')
+            >>> model = FSMTModel(config)
+
+        """
+        if "hidden_size" in common_kwargs:
+            raise ValueError("hidden size is called d_model")
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            decoder_start_token_id=decoder_start_token_id,
+            is_encoder_decoder=is_encoder_decoder,
+            tie_word_embeddings=tie_word_embeddings,
+            **common_kwargs,
+        )
+        self.langs = langs
+        self.src_vocab_size = src_vocab_size
+        self.tgt_vocab_size = tgt_vocab_size
+        self.d_model = d_model  # encoder_embed_dim and decoder_embed_dim
+        self.max_length = max_length
+
+        self.encoder_ffn_dim = encoder_ffn_dim
+        self.encoder_layers = self.num_hidden_layers = encoder_layers
+        self.encoder_attention_heads = encoder_attention_heads
+        self.encoder_layerdrop = encoder_layerdrop
+        self.decoder_layerdrop = decoder_layerdrop
+        self.decoder_ffn_dim = decoder_ffn_dim
+        self.decoder_layers = decoder_layers
+        self.decoder_attention_heads = decoder_attention_heads
+        self.max_position_embeddings = max_position_embeddings
+        self.init_std = init_std  # Normal(0, this parameter)
+        self.activation_function = activation_function
+
+        self.num_beams = num_beams
+        self.length_penalty = length_penalty
+        self.early_stopping = early_stopping
+
+        self.decoder = DecoderConfig(vocab_size=tgt_vocab_size, bos_token_id=eos_token_id)
+
+        self.scale_embedding = scale_embedding  # scale factor will be sqrt(d_model) if True
+
+        # 3 Types of Dropout
+        self.attention_dropout = attention_dropout
+        self.activation_dropout = activation_dropout
+        self.dropout = dropout
+
+    @property
+    def num_attention_heads(self) -> int:
+        return self.encoder_attention_heads
+
+    @property
+    def hidden_size(self) -> int:
+        return self.d_model
+
+    def to_dict(self):
+        """
+        Serializes this instance to a Python dictionary. Override the default `to_dict()` from `PretrainedConfig`.
+
+        Returns:
+            :obj:`Dict[str, any]`: Dictionary of all the attributes that make up this configuration instance,
+        """
+        output = copy.deepcopy(self.__dict__)
+        output["decoder"] = self.decoder.to_dict()
+        output["model_type"] = self.__class__.model_type
+        return output

src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py

@@ -0,0 +1,271 @@
+# coding=utf-8
+# Copyright 2018 The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Note: if you intend to run this script make sure you look under scripts/fsmt/
+# to locate the appropriate script to do the work correctly. There is a set of scripts to:
+# - download and prepare data and run the conversion script
+# - perform eval to get the best hparam into the config
+# - generate model_cards - useful if you have multiple models from the same paper
+
+import argparse
+import json
+import os
+import re
+from collections import OrderedDict
+from os.path import basename, dirname
+
+import fairseq
+import torch
+from fairseq import hub_utils
+from fairseq.data.dictionary import Dictionary
+
+from transformers import WEIGHTS_NAME, logging
+from transformers.configuration_fsmt import FSMTConfig
+from transformers.modeling_fsmt import FSMTForConditionalGeneration
+from transformers.tokenization_fsmt import VOCAB_FILES_NAMES
+from transformers.tokenization_utils_base import TOKENIZER_CONFIG_FILE
+
+
+logging.set_verbosity_warning()
+
+json_indent = 2
+
+# based on the results of a search on a range of `num_beams`, `length_penalty` and `early_stopping`
+# values against wmt19 test data to obtain the best BLEU scores, we will use the following defaults:
+#
+# * `num_beams`: 5 (higher scores better, but requires more memory/is slower, can be adjusted by users)
+# * `early_stopping`: `False` consistently scored better
+# * `length_penalty` varied, so will assign the best one depending on the model
+best_score_hparams = {
+    # fairseq:
+    "wmt19-ru-en": {"length_penalty": 1.1},
+    "wmt19-en-ru": {"length_penalty": 1.15},
+    "wmt19-en-de": {"length_penalty": 1.0},
+    "wmt19-de-en": {"length_penalty": 1.1},
+    # allenai:
+    "wmt16-en-de-dist-12-1": {"length_penalty": 0.6},
+    "wmt16-en-de-dist-6-1": {"length_penalty": 0.6},
+    "wmt16-en-de-12-1": {"length_penalty": 0.8},
+    "wmt19-de-en-6-6-base": {"length_penalty": 0.6},
+    "wmt19-de-en-6-6-big": {"length_penalty": 0.6},
+}
+
+# this remaps the different models to their organization names
+org_names = {}
+for m in ["wmt19-ru-en", "wmt19-en-ru", "wmt19-en-de", "wmt19-de-en"]:
+    org_names[m] = "facebook"
+for m in [
+    "wmt16-en-de-dist-12-1",
+    "wmt16-en-de-dist-6-1",
+    "wmt16-en-de-12-1",
+    "wmt19-de-en-6-6-base",
+    "wmt19-de-en-6-6-big",
+]:
+    org_names[m] = "allenai"
+
+
+def rewrite_dict_keys(d):
+    # (1) remove word breaking symbol, (2) add word ending symbol where the word is not broken up,
+    # e.g.: d = {'le@@': 5, 'tt@@': 6, 'er': 7} => {'le': 5, 'tt': 6, 'er</w>': 7}
+    d2 = dict((re.sub(r"@@$", "", k), v) if k.endswith("@@") else (re.sub(r"$", "</w>", k), v) for k, v in d.items())
+    keep_keys = "<s> <pad> </s> <unk>".split()
+    # restore the special tokens
+    for k in keep_keys:
+        del d2[f"{k}</w>"]
+        d2[k] = d[k]  # restore
+    return d2
+
+
+def convert_fsmt_checkpoint_to_pytorch(fsmt_checkpoint_path, pytorch_dump_folder_path):
+
+    # prep
+    assert os.path.exists(fsmt_checkpoint_path)
+    os.makedirs(pytorch_dump_folder_path, exist_ok=True)
+    print(f"Writing results to {pytorch_dump_folder_path}")
+
+    # handle various types of models
+
+    checkpoint_file = basename(fsmt_checkpoint_path)
+    fsmt_folder_path = dirname(fsmt_checkpoint_path)
+
+    cls = fairseq.model_parallel.models.transformer.ModelParallelTransformerModel
+    models = cls.hub_models()
+    kwargs = {"bpe": "fastbpe", "tokenizer": "moses"}
+    data_name_or_path = "."
+    # note: since the model dump is old, fairseq has upgraded its model some
+    # time later, and it does a whole lot of rewrites and splits on the saved
+    # weights, therefore we can't use torch.load() directly on the model file.
+    # see: upgrade_state_dict(state_dict) in fairseq_model.py
+    print(f"using checkpoint {checkpoint_file}")
+    chkpt = hub_utils.from_pretrained(
+        fsmt_folder_path, checkpoint_file, data_name_or_path, archive_map=models, **kwargs
+    )
+
+    args = dict(vars(chkpt["args"]))
+
+    src_lang = args["source_lang"]
+    tgt_lang = args["target_lang"]
+
+    data_root = dirname(pytorch_dump_folder_path)
+    model_dir = basename(pytorch_dump_folder_path)
+
+    # dicts
+    src_dict_file = os.path.join(fsmt_folder_path, f"dict.{src_lang}.txt")
+    tgt_dict_file = os.path.join(fsmt_folder_path, f"dict.{tgt_lang}.txt")
+
+    src_dict = Dictionary.load(src_dict_file)
+    src_vocab = rewrite_dict_keys(src_dict.indices)
+    src_vocab_size = len(src_vocab)
+    src_vocab_file = os.path.join(pytorch_dump_folder_path, "vocab-src.json")
+    print(f"Generating {src_vocab_file}")
+    with open(src_vocab_file, "w", encoding="utf-8") as f:
+        f.write(json.dumps(src_vocab, ensure_ascii=False, indent=json_indent))
+
+    tgt_dict = Dictionary.load(tgt_dict_file)
+    tgt_vocab = rewrite_dict_keys(tgt_dict.indices)
+    tgt_vocab_size = len(tgt_vocab)
+    tgt_vocab_file = os.path.join(pytorch_dump_folder_path, "vocab-tgt.json")
+    print(f"Generating {tgt_vocab_file}")
+    with open(tgt_vocab_file, "w", encoding="utf-8") as f:
+        f.write(json.dumps(tgt_vocab, ensure_ascii=False, indent=json_indent))
+
+    # merges_file (bpecodes)
+    merges_file = os.path.join(pytorch_dump_folder_path, VOCAB_FILES_NAMES["merges_file"])
+    fsmt_merges_file = os.path.join(fsmt_folder_path, "bpecodes")
+    with open(fsmt_merges_file, encoding="utf-8") as fin:
+        merges = fin.read()
+    merges = re.sub(r" \d+$", "", merges, 0, re.M)  # remove frequency number
+    print(f"Generating {merges_file}")
+    with open(merges_file, "w", encoding="utf-8") as fout:
+        fout.write(merges)
+
+    # model config
+    fsmt_model_config_file = os.path.join(pytorch_dump_folder_path, "config.json")
+
+    # validate bpe/tokenizer config, as currently it's hardcoded to moses+fastbpe -
+    # may have to modify the tokenizer if a different type is used by a future model
+    assert args["bpe"] == "fastbpe", f"need to extend tokenizer to support bpe={args['bpe']}"
+    assert args["tokenizer"] == "moses", f"need to extend tokenizer to support bpe={args['tokenizer']}"
+
+    model_conf = {
+        "architectures": ["FSMTForConditionalGeneration"],
+        "model_type": "fsmt",
+        "activation_dropout": args["activation_dropout"],
+        "activation_function": "relu",
+        "attention_dropout": args["attention_dropout"],
+        "d_model": args["decoder_embed_dim"],
+        "dropout": args["dropout"],
+        "init_std": 0.02,
+        "max_position_embeddings": args["max_source_positions"],
+        "num_hidden_layers": args["encoder_layers"],
+        "src_vocab_size": src_vocab_size,
+        "tgt_vocab_size": tgt_vocab_size,
+        "langs": [src_lang, tgt_lang],
+        "encoder_attention_heads": args["encoder_attention_heads"],
+        "encoder_ffn_dim": args["encoder_ffn_embed_dim"],
+        "encoder_layerdrop": args["encoder_layerdrop"],
+        "encoder_layers": args["encoder_layers"],
+        "decoder_attention_heads": args["decoder_attention_heads"],
+        "decoder_ffn_dim": args["decoder_ffn_embed_dim"],
+        "decoder_layerdrop": args["decoder_layerdrop"],
+        "decoder_layers": args["decoder_layers"],
+        "bos_token_id": 0,
+        "pad_token_id": 1,
+        "eos_token_id": 2,
+        "is_encoder_decoder": True,
+        "scale_embedding": not args["no_scale_embedding"],
+        "tie_word_embeddings": args["share_all_embeddings"],
+    }
+
+    # good hparam defaults to start with
+    model_conf["num_beams"] = 5
+    model_conf["early_stopping"] = False
+    if model_dir in best_score_hparams and "length_penalty" in best_score_hparams[model_dir]:
+        model_conf["length_penalty"] = best_score_hparams[model_dir]["length_penalty"]
+    else:
+        model_conf["length_penalty"] = 1.0
+
+    print(f"Generating {fsmt_model_config_file}")
+    with open(fsmt_model_config_file, "w", encoding="utf-8") as f:
+        f.write(json.dumps(model_conf, ensure_ascii=False, indent=json_indent))
+
+    # tokenizer config
+    fsmt_tokenizer_config_file = os.path.join(pytorch_dump_folder_path, TOKENIZER_CONFIG_FILE)
+
+    tokenizer_conf = {
+        "langs": [src_lang, tgt_lang],
+        "model_max_length": 1024,
+    }
+
+    print(f"Generating {fsmt_tokenizer_config_file}")
+    with open(fsmt_tokenizer_config_file, "w", encoding="utf-8") as f:
+        f.write(json.dumps(tokenizer_conf, ensure_ascii=False, indent=json_indent))
+
+    # model
+    model = chkpt["models"][0]
+    model_state_dict = model.state_dict()
+
+    # rename keys to start with 'model.'
+    model_state_dict = OrderedDict(("model." + k, v) for k, v in model_state_dict.items())
+
+    # remove unneeded keys
+    ignore_keys = [
+        "model.model",
+        "model.encoder.version",
+        "model.decoder.version",
+        "model.encoder_embed_tokens.weight",
+        "model.decoder_embed_tokens.weight",
+        "model.encoder.embed_positions._float_tensor",
+        "model.decoder.embed_positions._float_tensor",
+    ]
+    for k in ignore_keys:
+        model_state_dict.pop(k, None)
+
+    config = FSMTConfig.from_pretrained(pytorch_dump_folder_path)
+    model_new = FSMTForConditionalGeneration(config)
+
+    # check that it loads ok
+    model_new.load_state_dict(model_state_dict, strict=False)
+
+    # save
+    pytorch_weights_dump_path = os.path.join(pytorch_dump_folder_path, WEIGHTS_NAME)
+    print(f"Generating {pytorch_weights_dump_path}")
+    torch.save(model_state_dict, pytorch_weights_dump_path)
+
+    print("Conversion is done!")
+    print("\nLast step is to upload the files to s3")
+    print(f"cd {data_root}")
+    print(f"transformers-cli upload {model_dir}")
+    print(
+        "Note: CDN caches files for up to 24h, so either use a local model path "
+        "or use `from_pretrained(mname, use_cdn=False)` to use the non-cached version."
+    )
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    # Required parameters
+    parser.add_argument(
+        "--fsmt_checkpoint_path",
+        default=None,
+        type=str,
+        required=True,
+        help="Path to the official PyTorch checkpoint file which is expected to reside in the dump dir with dicts, bpecodes, etc.",
+    )
+    parser.add_argument(
+        "--pytorch_dump_folder_path", default=None, type=str, required=True, help="Path to the output PyTorch model."
+    )
+    args = parser.parse_args()
+    convert_fsmt_checkpoint_to_pytorch(args.fsmt_checkpoint_path, args.pytorch_dump_folder_path)

src/transformers/data/data_collator.py

@@ -434,13 +434,15 @@ class DataCollatorForNextSentencePrediction:
         else:
             input_ids = self._tensorize_batch(input_ids)
 
-        return {
+        result = {
             "input_ids": input_ids,
             "attention_mask": self._tensorize_batch(attention_masks),
             "token_type_ids": self._tensorize_batch(segment_ids),
-            "masked_lm_labels": mlm_labels if self.mlm else None,
             "next_sentence_label": torch.tensor(nsp_labels),
         }
+        if self.mlm:
+            result["masked_lm_labels"] = mlm_labels
+        return result
 
     def _tensorize_batch(self, examples: List[torch.Tensor]) -> torch.Tensor:
         length_of_first = examples[0].size(0)

src/transformers/integrations.py

@@ -3,8 +3,8 @@ import os
 
 import numpy as np
 
-from transformers.trainer_utils import PREFIX_CHECKPOINT_DIR, BestRun
-from transformers.utils import logging
+from .trainer_utils import PREFIX_CHECKPOINT_DIR, BestRun
+from .utils import logging
 
 
 logger = logging.get_logger(__name__)

src/transformers/modeling_auto.py

@@ -30,6 +30,7 @@ from .configuration_auto import (
     ElectraConfig,
     EncoderDecoderConfig,
     FlaubertConfig,
+    FSMTConfig,
     FunnelConfig,
     GPT2Config,
     LongformerConfig,
@@ -113,6 +114,7 @@ from .modeling_flaubert import (
     FlaubertModel,
     FlaubertWithLMHeadModel,
 )
+from .modeling_fsmt import FSMTForConditionalGeneration, FSMTModel
 from .modeling_funnel import (
     FunnelForMaskedLM,
     FunnelForMultipleChoice,
@@ -211,6 +213,7 @@ MODEL_MAPPING = OrderedDict(
         (TransfoXLConfig, TransfoXLModel),
         (XLNetConfig, XLNetModel),
         (FlaubertConfig, FlaubertModel),
+        (FSMTConfig, FSMTModel),
         (XLMConfig, XLMModel),
         (CTRLConfig, CTRLModel),
         (ElectraConfig, ElectraModel),
@@ -230,6 +233,7 @@ MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
         (CamembertConfig, CamembertForMaskedLM),
         (XLMRobertaConfig, XLMRobertaForMaskedLM),
         (BartConfig, BartForConditionalGeneration),
+        (FSMTConfig, FSMTForConditionalGeneration),
         (LongformerConfig, LongformerForMaskedLM),
         (RobertaConfig, RobertaForMaskedLM),
         (BertConfig, BertForPreTraining),
@@ -319,6 +323,7 @@ MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING = OrderedDict(
         (MarianConfig, MarianMTModel),
         (MBartConfig, MBartForConditionalGeneration),
         (BartConfig, BartForConditionalGeneration),
+        (FSMTConfig, FSMTForConditionalGeneration),
         (EncoderDecoderConfig, EncoderDecoderModel),
     ]
 )
@@ -519,10 +524,10 @@ class AutoModel:
 
         Examples::
 
-            from transformers import AutoConfig, AutoModel
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = AutoModel.from_config(config)
+            >>> from transformers import AutoConfig, AutoModel
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = AutoModel.from_config(config)
         """
         for config_class, model_class in MODEL_MAPPING.items():
             if isinstance(config, config_class):
@@ -613,10 +618,10 @@ class AutoModelForPreTraining:
 
         Examples::
 
-            from transformers import AutoConfig, AutoModelForPreTraining
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = AutoModelForPreTraining.from_config(config)
+            >>> from transformers import AutoConfig, AutoModelForPreTraining
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = AutoModelForPreTraining.from_config(config)
         """
         for config_class, model_class in MODEL_FOR_PRETRAINING_MAPPING.items():
             if isinstance(config, config_class):
@@ -713,10 +718,10 @@ class AutoModelWithLMHead:
 
         Examples::
 
-            from transformers import AutoConfig, AutoModelWithLMHead
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = AutoModelWithLMHead.from_config(config)
+            >>> from transformers import AutoConfig, AutoModelWithLMHead
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = AutoModelWithLMHead.from_config(config)
         """
         warnings.warn(
             "The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use "
@@ -819,10 +824,10 @@ class AutoModelForCausalLM:
 
         Examples::
 
-            from transformers import AutoConfig, AutoModelForCausalLM
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('gpt2')
-            model = AutoModelForCausalLM.from_config(config)
+            >>> from transformers import AutoConfig, AutoModelForCausalLM
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('gpt2')
+            >>> model = AutoModelForCausalLM.from_config(config)
         """
         for config_class, model_class in MODEL_FOR_CAUSAL_LM_MAPPING.items():
             if isinstance(config, config_class):
@@ -913,10 +918,10 @@ class AutoModelForMaskedLM:
 
         Examples::
 
-            from transformers import AutoConfig, AutoModelForMaskedLM
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = AutoModelForMaskedLM.from_config(config)
+            >>> from transformers import AutoConfig, AutoModelForMaskedLM
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = AutoModelForMaskedLM.from_config(config)
         """
         for config_class, model_class in MODEL_FOR_MASKED_LM_MAPPING.items():
             if isinstance(config, config_class):
@@ -1007,10 +1012,10 @@ class AutoModelForSeq2SeqLM:
 
         Examples::
 
-            from transformers import AutoConfig, AutoModelForSeq2SeqLM
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('t5')
-            model = AutoModelForSeq2SeqLM.from_config(config)
+            >>> from transformers import AutoConfig, AutoModelForSeq2SeqLM
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('t5')
+            >>> model = AutoModelForSeq2SeqLM.from_config(config)
         """
         for config_class, model_class in MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING.items():
             if isinstance(config, config_class):
@@ -1105,10 +1110,10 @@ class AutoModelForSequenceClassification:
 
         Examples::
 
-            from transformers import AutoConfig, AutoModelForSequenceClassification
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = AutoModelForSequenceClassification.from_config(config)
+            >>> from transformers import AutoConfig, AutoModelForSequenceClassification
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = AutoModelForSequenceClassification.from_config(config)
         """
         for config_class, model_class in MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING.items():
             if isinstance(config, config_class):
@@ -1202,10 +1207,10 @@ class AutoModelForQuestionAnswering:
 
         Examples::
 
-            from transformers import AutoConfig, AutoModelForQuestionAnswering
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = AutoModelForQuestionAnswering.from_config(config)
+            >>> from transformers import AutoConfig, AutoModelForQuestionAnswering
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = AutoModelForQuestionAnswering.from_config(config)
         """
         for config_class, model_class in MODEL_FOR_QUESTION_ANSWERING_MAPPING.items():
             if isinstance(config, config_class):
@@ -1301,10 +1306,10 @@ class AutoModelForTokenClassification:
 
         Examples::
 
-            from transformers import AutoConfig, AutoModelForTokenClassification
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = AutoModelForTokenClassification.from_config(config)
+            >>> from transformers import AutoConfig, AutoModelForTokenClassification
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = AutoModelForTokenClassification.from_config(config)
         """
         for config_class, model_class in MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING.items():
             if isinstance(config, config_class):
@@ -1401,10 +1406,10 @@ class AutoModelForMultipleChoice:
 
         Examples::
 
-            from transformers import AutoConfig, AutoModelForMultipleChoice
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = AutoModelForMultipleChoice.from_config(config)
+            >>> from transformers import AutoConfig, AutoModelForMultipleChoice
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = AutoModelForMultipleChoice.from_config(config)
         """
         for config_class, model_class in MODEL_FOR_MULTIPLE_CHOICE_MAPPING.items():
             if isinstance(config, config_class):

src/transformers/modeling_bart.py

@@ -76,18 +76,18 @@ BART_START_DOCSTRING = r"""
 BART_GENERATION_EXAMPLE = r"""
     Summarization example::
 
-        from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
+        >>> from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
 
-        # see ``examples/summarization/bart/run_eval.py`` for a longer example
-        model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')
-        tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')
+        >>> # see ``examples/summarization/bart/run_eval.py`` for a longer example
+        >>> model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')
+        >>> tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')
 
-        ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
-        inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')
+        >>> ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
+        >>> inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')
 
-        # Generate Summary
-        summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=5, early_stopping=True)
-        print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids])
+        >>> # Generate Summary
+        >>> summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=5, early_stopping=True)
+        >>> print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids])
 
 """
 
@@ -1023,21 +1023,21 @@ class BartForConditionalGeneration(PretrainedBartModel):
 
         Conditional generation example::
 
-                # Mask filling only works for bart-large
-                from transformers import BartTokenizer, BartForConditionalGeneration
-                tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
-                TXT = "My friends are <mask> but they eat too many carbs."
+                >>> # Mask filling only works for bart-large
+                >>> from transformers import BartTokenizer, BartForConditionalGeneration
+                >>> tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
+                >>> TXT = "My friends are <mask> but they eat too many carbs."
 
-                model = BartForConditionalGeneration.from_pretrained('facebook/bart-large')
-                input_ids = tokenizer([TXT], return_tensors='pt')['input_ids']
-                logits = model(input_ids).logits
+                >>> model = BartForConditionalGeneration.from_pretrained('facebook/bart-large')
+                >>> input_ids = tokenizer([TXT], return_tensors='pt')['input_ids']
+                >>> logits = model(input_ids).logits
 
-                masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()
-                probs = logits[0, masked_index].softmax(dim=0)
-                values, predictions = probs.topk(5)
+                >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()
+                >>> probs = logits[0, masked_index].softmax(dim=0)
+                >>> values, predictions = probs.topk(5)
 
-                tokenizer.decode(predictions).split()
-                # ['good', 'great', 'all', 'really', 'very']
+                >>> tokenizer.decode(predictions).split()
+                >>> # ['good', 'great', 'all', 'really', 'very']
         """
         if "lm_labels" in unused:
             warnings.warn(

src/transformers/modeling_dpr.py

@@ -425,11 +425,11 @@ class DPRContextEncoder(DPRPretrainedContextEncoder):
 
         Examples::
 
-            from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
-            tokenizer = DPRContextEncoderTokenizer.from_pretrained('facebook/dpr-ctx_encoder-single-nq-base')
-            model = DPRContextEncoder.from_pretrained('facebook/dpr-ctx_encoder-single-nq-base', return_dict=True)
-            input_ids = tokenizer("Hello, is my dog cute ?", return_tensors='pt')["input_ids"]
-            embeddings = model(input_ids).pooler_output
+            >>> from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
+            >>> tokenizer = DPRContextEncoderTokenizer.from_pretrained('facebook/dpr-ctx_encoder-single-nq-base')
+            >>> model = DPRContextEncoder.from_pretrained('facebook/dpr-ctx_encoder-single-nq-base', return_dict=True)
+            >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors='pt')["input_ids"]
+            >>> embeddings = model(input_ids).pooler_output
         """
 
         output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
@@ -503,11 +503,11 @@ class DPRQuestionEncoder(DPRPretrainedQuestionEncoder):
 
         Examples::
 
-            from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer
-            tokenizer = DPRQuestionEncoderTokenizer.from_pretrained('facebook/dpr-question_encoder-single-nq-base')
-            model = DPRQuestionEncoder.from_pretrained('facebook/dpr-question_encoder-single-nq-base', return_dict=True)
-            input_ids = tokenizer("Hello, is my dog cute ?", return_tensors='pt')["input_ids"]
-            embeddings = model(input_ids).pooler_output
+            >>> from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer
+            >>> tokenizer = DPRQuestionEncoderTokenizer.from_pretrained('facebook/dpr-question_encoder-single-nq-base')
+            >>> model = DPRQuestionEncoder.from_pretrained('facebook/dpr-question_encoder-single-nq-base', return_dict=True)
+            >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors='pt')["input_ids"]
+            >>> embeddings = model(input_ids).pooler_output
         """
         output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
         output_hidden_states = (
@@ -579,19 +579,19 @@ class DPRReader(DPRPretrainedReader):
 
         Examples::
 
-            from transformers import DPRReader, DPRReaderTokenizer
-            tokenizer = DPRReaderTokenizer.from_pretrained('facebook/dpr-reader-single-nq-base')
-            model = DPRReader.from_pretrained('facebook/dpr-reader-single-nq-base', return_dict=True)
-            encoded_inputs = tokenizer(
-                    questions=["What is love ?"],
-                    titles=["Haddaway"],
-                    texts=["'What Is Love' is a song recorded by the artist Haddaway"],
-                    return_tensors='pt'
-                )
-            outputs = model(**encoded_inputs)
-            start_logits = outputs.stat_logits
-            end_logits = outputs.end_logits
-            relevance_logits = outputs.relevance_logits
+            >>> from transformers import DPRReader, DPRReaderTokenizer
+            >>> tokenizer = DPRReaderTokenizer.from_pretrained('facebook/dpr-reader-single-nq-base')
+            >>> model = DPRReader.from_pretrained('facebook/dpr-reader-single-nq-base', return_dict=True)
+            >>> encoded_inputs = tokenizer(
+            ...         questions=["What is love ?"],
+            ...         titles=["Haddaway"],
+            ...         texts=["'What Is Love' is a song recorded by the artist Haddaway"],
+            ...         return_tensors='pt'
+            ...     )
+            >>> outputs = model(**encoded_inputs)
+            >>> start_logits = outputs.stat_logits
+            >>> end_logits = outputs.end_logits
+            >>> relevance_logits = outputs.relevance_logits
 
         """
         output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions

src/transformers/modeling_openai.py

@@ -655,21 +655,21 @@ class OpenAIGPTDoubleHeadsModel(OpenAIGPTPreTrainedModel):
 
         Examples::
 
-            from transformers import OpenAIGPTTokenizer, OpenAIGPTDoubleHeadsModel
-            import torch
+            >>> from transformers import OpenAIGPTTokenizer, OpenAIGPTDoubleHeadsModel
+            >>> import torch
 
-            tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')
-            model = OpenAIGPTDoubleHeadsModel.from_pretrained('openai-gpt', return_dict=True)
-            tokenizer.add_special_tokens({'cls_token': '[CLS]'})  # Add a [CLS] to the vocabulary (we should train it also!)
-            model.resize_token_embeddings(len(tokenizer))
+            >>> tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')
+            >>> model = OpenAIGPTDoubleHeadsModel.from_pretrained('openai-gpt', return_dict=True)
+            >>> tokenizer.add_special_tokens({'cls_token': '[CLS]'})  # Add a [CLS] to the vocabulary (we should train it also!)
+            >>> model.resize_token_embeddings(len(tokenizer))
 
-            choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"]
-            input_ids = torch.tensor([tokenizer.encode(s) for s in choices]).unsqueeze(0)  # Batch size 1, 2 choices
-            mc_token_ids = torch.tensor([input_ids.size(-1)-1, input_ids.size(-1)-1]).unsqueeze(0)  # Batch size 1
+            >>> choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"]
+            >>> input_ids = torch.tensor([tokenizer.encode(s) for s in choices]).unsqueeze(0)  # Batch size 1, 2 choices
+            >>> mc_token_ids = torch.tensor([input_ids.size(-1)-1, input_ids.size(-1)-1]).unsqueeze(0)  # Batch size 1
 
-            outputs = model(input_ids, mc_token_ids=mc_token_ids)
-            lm_logits = outputs.lm_logits
-            mc_logits = outputs.mc_logits
+            >>> outputs = model(input_ids, mc_token_ids=mc_token_ids)
+            >>> lm_logits = outputs.lm_logits
+            >>> mc_logits = outputs.mc_logits
         """
         return_dict = return_dict if return_dict is not None else self.config.use_return_dict
         if "lm_labels" in kwargs:

src/transformers/modeling_roberta.py

@@ -303,6 +303,7 @@ class RobertaForCausalLM(BertPreTrainedModel):
 class RobertaForMaskedLM(BertPreTrainedModel):
     config_class = RobertaConfig
     base_model_prefix = "roberta"
+    authorized_missing_keys = [r"position_ids", r"lm_head\.decoder\.bias"]
 
     def __init__(self, config):
         super().__init__(config)

src/transformers/modeling_tf_albert.py

@@ -827,13 +827,13 @@ class TFAlbertForPreTraining(TFAlbertPreTrainedModel):
         Return:
 
         Examples::
-            import tensorflow as tf
-            from transformers import AlbertTokenizer, TFAlbertForPreTraining
-            tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
-            model = TFAlbertForPreTraining.from_pretrained('albert-base-v2')
-            input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :]  # Batch size 1
-            outputs = model(input_ids)
-            prediction_scores, sop_scores = outputs[:2]
+            >>> import tensorflow as tf
+            >>> from transformers import AlbertTokenizer, TFAlbertForPreTraining
+            >>> tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
+            >>> model = TFAlbertForPreTraining.from_pretrained('albert-base-v2')
+            >>> input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :]  # Batch size 1
+            >>> outputs = model(input_ids)
+            >>> prediction_scores, sop_scores = outputs[:2]
         """
         return_dict = kwargs.get("return_dict")
         return_dict = return_dict if return_dict is not None else self.albert.return_dict

src/transformers/modeling_tf_auto.py

@@ -448,10 +448,10 @@ class TFAutoModel(object):
 
         Examples::
 
-            from transformers import AutoConfig, TFAutoModel
-            # Download configuration from S3 and cache.
-            config = TFAutoConfig.from_pretrained('bert-base-uncased')
-            model = TFAutoModel.from_config(config)
+            >>> from transformers import AutoConfig, TFAutoModel
+            >>> # Download configuration from S3 and cache.
+            >>> config = TFAutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = TFAutoModel.from_config(config)
         """
         for config_class, model_class in TF_MODEL_MAPPING.items():
             if isinstance(config, config_class):
@@ -542,10 +542,10 @@ class TFAutoModelForPreTraining(object):
 
         Examples::
 
-            from transformers import AutoConfig, TFAutoModelForPreTraining
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = TFAutoModelForPreTraining.from_config(config)
+            >>> from transformers import AutoConfig, TFAutoModelForPreTraining
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = TFAutoModelForPreTraining.from_config(config)
         """
         for config_class, model_class in TF_MODEL_FOR_PRETRAINING_MAPPING.items():
             if isinstance(config, config_class):
@@ -642,10 +642,10 @@ class TFAutoModelWithLMHead(object):
 
         Examples::
 
-            from transformers import AutoConfig, TFAutoModelWithLMHead
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = TFAutoModelWithLMHead.from_config(config)
+            >>> from transformers import AutoConfig, TFAutoModelWithLMHead
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = TFAutoModelWithLMHead.from_config(config)
         """
         warnings.warn(
             "The class `TFAutoModelWithLMHead` is deprecated and will be removed in a future version. Please use "
@@ -750,10 +750,10 @@ class TFAutoModelForCausalLM:
 
         Examples::
 
-            from transformers import AutoConfig, TFAutoModelForCausalLM
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('gpt2')
-            model = TFAutoModelForCausalLM.from_config(config)
+            >>> from transformers import AutoConfig, TFAutoModelForCausalLM
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('gpt2')
+            >>> model = TFAutoModelForCausalLM.from_config(config)
         """
         for config_class, model_class in TF_MODEL_FOR_CAUSAL_LM_MAPPING.items():
             if isinstance(config, config_class):
@@ -844,10 +844,10 @@ class TFAutoModelForMaskedLM:
 
         Examples::
 
-            from transformers import AutoConfig, TFAutoModelForMaskedLM
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = TFAutoModelForMaskedLM.from_config(config)
+            >>> from transformers import AutoConfig, TFAutoModelForMaskedLM
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = TFAutoModelForMaskedLM.from_config(config)
         """
         for config_class, model_class in TF_MODEL_FOR_MASKED_LM_MAPPING.items():
             if isinstance(config, config_class):
@@ -938,10 +938,10 @@ class TFAutoModelForSeq2SeqLM:
 
         Examples::
 
-            from transformers import AutoConfig, TFAutoModelForSeq2SeqLM
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('t5')
-            model = TFAutoModelForSeq2SeqLM.from_config(config)
+            >>> from transformers import AutoConfig, TFAutoModelForSeq2SeqLM
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('t5')
+            >>> model = TFAutoModelForSeq2SeqLM.from_config(config)
         """
         for config_class, model_class in TF_MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING.items():
             if isinstance(config, config_class):
@@ -1036,10 +1036,10 @@ class TFAutoModelForSequenceClassification(object):
 
         Examples::
 
-            from transformers import AutoConfig, TFAutoModelForSequenceClassification
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = TFAutoModelForSequenceClassification.from_config(config)
+            >>> from transformers import AutoConfig, TFAutoModelForSequenceClassification
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = TFAutoModelForSequenceClassification.from_config(config)
         """
         for config_class, model_class in TF_MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING.items():
             if isinstance(config, config_class):
@@ -1133,10 +1133,10 @@ class TFAutoModelForQuestionAnswering(object):
 
         Examples::
 
-            from transformers import AutoConfig, TFAutoModelForQuestionAnswering
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = TFAutoModelForQuestionAnswering.from_config(config)
+            >>> from transformers import AutoConfig, TFAutoModelForQuestionAnswering
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = TFAutoModelForQuestionAnswering.from_config(config)
         """
         for config_class, model_class in TF_MODEL_FOR_QUESTION_ANSWERING_MAPPING.items():
             if isinstance(config, config_class):
@@ -1230,10 +1230,10 @@ class TFAutoModelForTokenClassification:
 
         Examples::
 
-            from transformers import AutoConfig, TFAutoModelForTokenClassification
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = TFAutoModelForTokenClassification.from_config(config)
+            >>> from transformers import AutoConfig, TFAutoModelForTokenClassification
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = TFAutoModelForTokenClassification.from_config(config)
         """
         for config_class, model_class in TF_MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING.items():
             if isinstance(config, config_class):
@@ -1328,10 +1328,10 @@ class TFAutoModelForMultipleChoice:
 
         Examples::
 
-            from transformers import AutoConfig, TFAutoModelForMultipleChoice
-            # Download configuration from S3 and cache.
-            config = AutoConfig.from_pretrained('bert-base-uncased')
-            model = TFAutoModelForMultipleChoice.from_config(config)
+            >>> from transformers import AutoConfig, TFAutoModelForMultipleChoice
+            >>> # Download configuration from S3 and cache.
+            >>> config = AutoConfig.from_pretrained('bert-base-uncased')
+            >>> model = TFAutoModelForMultipleChoice.from_config(config)
         """
         for config_class, model_class in TF_MODEL_FOR_MULTIPLE_CHOICE_MAPPING.items():
             if isinstance(config, config_class):

src/transformers/modeling_tf_bert.py

@@ -792,14 +792,14 @@ class TFBertForPreTraining(TFBertPreTrainedModel):
 
         Examples::
 
-            import tensorflow as tf
-            from transformers import BertTokenizer, TFBertForPreTraining
+            >>> import tensorflow as tf
+            >>> from transformers import BertTokenizer, TFBertForPreTraining
 
-            tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
-            model = TFBertForPreTraining.from_pretrained('bert-base-uncased')
-            input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :]  # Batch size 1
-            outputs = model(input_ids)
-            prediction_scores, seq_relationship_scores = outputs[:2]
+            >>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
+            >>> model = TFBertForPreTraining.from_pretrained('bert-base-uncased')
+            >>> input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :]  # Batch size 1
+            >>> outputs = model(input_ids)
+            >>> prediction_scores, seq_relationship_scores = outputs[:2]
 
         """
         return_dict = kwargs.get("return_dict")
@@ -1004,18 +1004,18 @@ class TFBertForNextSentencePrediction(TFBertPreTrainedModel):
 
         Examples::
 
-            import tensorflow as tf
-            from transformers import BertTokenizer, TFBertForNextSentencePrediction
+            >>> import tensorflow as tf
+            >>> from transformers import BertTokenizer, TFBertForNextSentencePrediction
 
-            tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
-            model = TFBertForNextSentencePrediction.from_pretrained('bert-base-uncased')
+            >>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
+            >>> model = TFBertForNextSentencePrediction.from_pretrained('bert-base-uncased')
 
-            prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
-            next_sentence = "The sky is blue due to the shorter wavelength of blue light."
-            encoding = tokenizer(prompt, next_sentence, return_tensors='tf')
+            >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
+            >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light."
+            >>> encoding = tokenizer(prompt, next_sentence, return_tensors='tf')
 
-            logits = model(encoding['input_ids'], token_type_ids=encoding['token_type_ids'])[0]
-            assert logits[0][0] < logits[0][1] # the next sentence was random
+            >>> logits = model(encoding['input_ids'], token_type_ids=encoding['token_type_ids'])[0]
+            >>> assert logits[0][0] < logits[0][1] # the next sentence was random
         """
         return_dict = kwargs.get("return_dict")
         return_dict = return_dict if return_dict is not None else self.bert.return_dict

src/transformers/modeling_tf_electra.py

@@ -3,9 +3,8 @@ from typing import Optional, Tuple
 
 import tensorflow as tf
 
-from transformers import ElectraConfig
-
 from .activations_tf import get_tf_activation
+from .configuration_electra import ElectraConfig
 from .file_utils import (
     MULTIPLE_CHOICE_DUMMY_INPUTS,
     ModelOutput,
@@ -501,14 +500,14 @@ class TFElectraForPreTraining(TFElectraPreTrainedModel):
 
         Examples::
 
-            import tensorflow as tf
-            from transformers import ElectraTokenizer, TFElectraForPreTraining
+            >>> import tensorflow as tf
+            >>> from transformers import ElectraTokenizer, TFElectraForPreTraining
 
-            tokenizer = ElectraTokenizer.from_pretrained('google/electra-small-discriminator')
-            model = TFElectraForPreTraining.from_pretrained('google/electra-small-discriminator')
-            input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
-            outputs = model(input_ids)
-            scores = outputs[0]
+            >>> tokenizer = ElectraTokenizer.from_pretrained('google/electra-small-discriminator')
+            >>> model = TFElectraForPreTraining.from_pretrained('google/electra-small-discriminator')
+            >>> input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
+            >>> outputs = model(input_ids)
+            >>> scores = outputs[0]
         """
         return_dict = return_dict if return_dict is not None else self.electra.config.return_dict
 

src/transformers/modeling_tf_lxmert.py

@@ -23,8 +23,6 @@ from typing import Dict, Optional, Tuple
 
 import tensorflow as tf
 
-from transformers import BatchEncoding
-
 from .activations_tf import get_tf_activation
 from .configuration_lxmert import LxmertConfig
 from .file_utils import (
@@ -35,6 +33,7 @@ from .file_utils import (
     replace_return_docstrings,
 )
 from .modeling_tf_utils import TFPreTrainedModel, get_initializer, keras_serializable, shape_list
+from .tokenization_utils_base import BatchEncoding
 
 
 logger = logging.getLogger(__name__)

src/transformers/modeling_tf_utils.py

@@ -504,17 +504,17 @@ class TFPreTrainedModel(tf.keras.Model, TFModelUtilsMixin, TFGenerationMixin):
 
         Examples::
 
-            from transformers import BertConfig, TFBertModel
-            # Download model and configuration from S3 and cache.
-            model = TFBertModel.from_pretrained('bert-base-uncased')
-            # Model was saved using `save_pretrained('./test/saved_model/')` (for example purposes, not runnable).
-            model = TFBertModel.from_pretrained('./test/saved_model/')
-            # Update configuration during loading.
-            model = TFBertModel.from_pretrained('bert-base-uncased', output_attentions=True)
-            assert model.config.output_attentions == True
-            # Loading from a Pytorch model file instead of a TensorFlow checkpoint (slower, for example purposes, not runnable).
-            config = BertConfig.from_json_file('./pt_model/my_pt_model_config.json')
-            model = TFBertModel.from_pretrained('./pt_model/my_pytorch_model.bin', from_pt=True, config=config)
+            >>> from transformers import BertConfig, TFBertModel
+            >>> # Download model and configuration from S3 and cache.
+            >>> model = TFBertModel.from_pretrained('bert-base-uncased')
+            >>> # Model was saved using `save_pretrained('./test/saved_model/')` (for example purposes, not runnable).
+            >>> model = TFBertModel.from_pretrained('./test/saved_model/')
+            >>> # Update configuration during loading.
+            >>> model = TFBertModel.from_pretrained('bert-base-uncased', output_attentions=True)
+            >>> assert model.config.output_attentions == True
+            >>> # Loading from a Pytorch model file instead of a TensorFlow checkpoint (slower, for example purposes, not runnable).
+            >>> config = BertConfig.from_json_file('./pt_model/my_pt_model_config.json')
+            >>> model = TFBertModel.from_pretrained('./pt_model/my_pytorch_model.bin', from_pt=True, config=config)
 
         """
         config = kwargs.pop("config", None)

src/transformers/modeling_tf_xlnet.py

@@ -1201,25 +1201,25 @@ class TFXLNetLMHeadModel(TFXLNetPreTrainedModel, TFCausalLanguageModelingLoss):
 
         Examples::
 
-            import tensorflow as tf
-            import numpy as np
-            from transformers import XLNetTokenizer, TFXLNetLMHeadModel
+            >>> import tensorflow as tf
+            >>> import numpy as np
+            >>> from transformers import XLNetTokenizer, TFXLNetLMHeadModel
 
-            tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')
-            model = TFXLNetLMHeadModel.from_pretrained('xlnet-large-cased')
+            >>> tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')
+            >>> model = TFXLNetLMHeadModel.from_pretrained('xlnet-large-cased')
 
-            # We show how to setup inputs to predict a next token using a bi-directional context.
-            input_ids = tf.constant(tokenizer.encode("Hello, my dog is very <mask>", add_special_tokens=True))[None, :]  # We will predict the masked token
+            >>> # We show how to setup inputs to predict a next token using a bi-directional context.
+            >>> input_ids = tf.constant(tokenizer.encode("Hello, my dog is very <mask>", add_special_tokens=True))[None, :]  # We will predict the masked token
 
-            perm_mask = np.zeros((1, input_ids.shape[1], input_ids.shape[1]))
-            perm_mask[:, :, -1] = 1.0  # Previous tokens don't see last token
+            >>> perm_mask = np.zeros((1, input_ids.shape[1], input_ids.shape[1]))
+            >>> perm_mask[:, :, -1] = 1.0  # Previous tokens don't see last token
 
-            target_mapping = np.zeros((1, 1, input_ids.shape[1]))  # Shape [1, 1, seq_length] => let's predict one token
-            target_mapping[0, 0, -1] = 1.0  # Our first (and only) prediction will be the last token of the sequence (the masked token)
+            >>> target_mapping = np.zeros((1, 1, input_ids.shape[1]))  # Shape [1, 1, seq_length] => let's predict one token
+            >>> target_mapping[0, 0, -1] = 1.0  # Our first (and only) prediction will be the last token of the sequence (the masked token)
 
-            outputs = model(input_ids, perm_mask=tf.constant(perm_mask, dtype=tf.float32), target_mapping=tf.constant(target_mapping, dtype=tf.float32))
+            >>> outputs = model(input_ids, perm_mask=tf.constant(perm_mask, dtype=tf.float32), target_mapping=tf.constant(target_mapping, dtype=tf.float32))
 
-            next_token_logits = outputs[0]  # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size]
+            >>> next_token_logits = outputs[0]  # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size]
 
         """
         return_dict = return_dict if return_dict is not None else self.transformer.return_dict

src/transformers/modeling_utils.py

@@ -391,10 +391,14 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin):
           derived classes of the same architecture adding modules on top of the base model.
         - **authorized_missing_keys** (:obj:`Optional[List[str]]`) -- A list of re pattern of tensor names to ignore
           when loading the model (and avoid unnecessary warnings).
+        - **keys_to_never_save** (:obj:`Optional[List[str]]`) -- A list of of tensor names to ignore
+          when saving the model (useful for keys that aren't trained, but which are deterministic)
+
     """
     config_class = None
     base_model_prefix = ""
     authorized_missing_keys = None
+    keys_to_never_save = None
 
     @property
     def dummy_inputs(self) -> Dict[str, torch.Tensor]:
@@ -688,6 +692,12 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin):
         # Attach architecture to the config
         model_to_save.config.architectures = [model_to_save.__class__.__name__]
 
+        state_dict = model_to_save.state_dict()
+
+        # Handle the case where some state_dict keys shouldn't be saved
+        if self.keys_to_never_save is not None:
+            state_dict = {k: v for k, v in state_dict.items() if k not in self.keys_to_never_save}
+
         # If we save using the predefined names, we can load using `from_pretrained`
         output_model_file = os.path.join(save_directory, WEIGHTS_NAME)
 
@@ -698,10 +708,10 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin):
                 # Save configuration file
                 model_to_save.config.save_pretrained(save_directory)
             # xm.save takes care of saving only from master
-            xm.save(model_to_save.state_dict(), output_model_file)
+            xm.save(state_dict, output_model_file)
         else:
             model_to_save.config.save_pretrained(save_directory)
-            torch.save(model_to_save.state_dict(), output_model_file)
+            torch.save(state_dict, output_model_file)
 
         logger.info("Model weights saved in {}".format(output_model_file))
 
@@ -804,17 +814,17 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin):
 
         Examples::
 
-            from transformers import BertConfig, BertModel
-            # Download model and configuration from S3 and cache.
-            model = BertModel.from_pretrained('bert-base-uncased')
-            # Model was saved using `save_pretrained('./test/saved_model/')` (for example purposes, not runnable).
-            model = BertModel.from_pretrained('./test/saved_model/')
-            # Update configuration during loading.
-            model = BertModel.from_pretrained('bert-base-uncased', output_attentions=True)
-            assert model.config.output_attentions == True
-            # Loading from a TF checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable).
-            config = BertConfig.from_json_file('./tf_model/my_tf_model_config.json')
-            model = BertModel.from_pretrained('./tf_model/my_tf_checkpoint.ckpt.index', from_tf=True, config=config)
+            >>> from transformers import BertConfig, BertModel
+            >>> # Download model and configuration from S3 and cache.
+            >>> model = BertModel.from_pretrained('bert-base-uncased')
+            >>> # Model was saved using `save_pretrained('./test/saved_model/')` (for example purposes, not runnable).
+            >>> model = BertModel.from_pretrained('./test/saved_model/')
+            >>> # Update configuration during loading.
+            >>> model = BertModel.from_pretrained('bert-base-uncased', output_attentions=True)
+            >>> assert model.config.output_attentions == True
+            >>> # Loading from a TF checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable).
+            >>> config = BertConfig.from_json_file('./tf_model/my_tf_model_config.json')
+            >>> model = BertModel.from_pretrained('./tf_model/my_tf_checkpoint.ckpt.index', from_tf=True, config=config)
         """
         config = kwargs.pop("config", None)
         state_dict = kwargs.pop("state_dict", None)
@@ -931,7 +941,7 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin):
             else:
                 # Load from our TensorFlow 2.0 checkpoints
                 try:
-                    from transformers import load_tf2_checkpoint_in_pytorch_model
+                    from .modeling_tf_pytorch_utils import load_tf2_checkpoint_in_pytorch_model
 
                     model = load_tf2_checkpoint_in_pytorch_model(model, resolved_archive_file, allow_missing_keys=True)
                 except ImportError:

src/transformers/modeling_xlnet.py

@@ -1313,7 +1313,7 @@ class XLNetLMHeadModel(XLNetPreTrainedModel):
         target_mapping = torch.zeros(
             (effective_batch_size, 1, sequence_length), dtype=torch.float, device=input_ids.device
         )
-        target_mapping[0, 0, -1] = 1.0
+        target_mapping[:, 0, -1] = 1.0
 
         inputs = {
             "input_ids": input_ids,
@@ -1360,34 +1360,34 @@ class XLNetLMHeadModel(XLNetPreTrainedModel):
 
         Examples::
 
-            from transformers import XLNetTokenizer, XLNetLMHeadModel
-            import torch
+            >>> from transformers import XLNetTokenizer, XLNetLMHeadModel
+            >>> import torch
 
-            tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')
-            model = XLNetLMHeadModel.from_pretrained('xlnet-large-cased', return_dict=True)
+            >>> tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')
+            >>> model = XLNetLMHeadModel.from_pretrained('xlnet-large-cased', return_dict=True)
 
-            # We show how to setup inputs to predict a next token using a bi-directional context.
-            input_ids = torch.tensor(tokenizer.encode("Hello, my dog is very <mask>", add_special_tokens=False)).unsqueeze(0)  # We will predict the masked token
-            perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float)
-            perm_mask[:, :, -1] = 1.0  # Previous tokens don't see last token
-            target_mapping = torch.zeros((1, 1, input_ids.shape[1]), dtype=torch.float)  # Shape [1, 1, seq_length] => let's predict one token
-            target_mapping[0, 0, -1] = 1.0  # Our first (and only) prediction will be the last token of the sequence (the masked token)
+            >>> # We show how to setup inputs to predict a next token using a bi-directional context.
+            >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is very <mask>", add_special_tokens=False)).unsqueeze(0)  # We will predict the masked token
+            >>> perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float)
+            >>> perm_mask[:, :, -1] = 1.0  # Previous tokens don't see last token
+            >>> target_mapping = torch.zeros((1, 1, input_ids.shape[1]), dtype=torch.float)  # Shape [1, 1, seq_length] => let's predict one token
+            >>> target_mapping[0, 0, -1] = 1.0  # Our first (and only) prediction will be the last token of the sequence (the masked token)
 
-            outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping)
-            next_token_logits = outputs[0]  # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size]
+            >>> outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping)
+            >>> next_token_logits = outputs[0]  # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size]
 
-            # The same way can the XLNetLMHeadModel be used to be trained by standard auto-regressive language modeling.
-            input_ids = torch.tensor(tokenizer.encode("Hello, my dog is very <mask>", add_special_tokens=False)).unsqueeze(0)  # We will predict the masked token
-            labels = torch.tensor(tokenizer.encode("cute", add_special_tokens=False)).unsqueeze(0)
-            assert labels.shape[0] == 1, 'only one word will be predicted'
-            perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float)
-            perm_mask[:, :, -1] = 1.0  # Previous tokens don't see last token as is done in standard auto-regressive lm training
-            target_mapping = torch.zeros((1, 1, input_ids.shape[1]), dtype=torch.float)  # Shape [1, 1, seq_length] => let's predict one token
-            target_mapping[0, 0, -1] = 1.0  # Our first (and only) prediction will be the last token of the sequence (the masked token)
+            >>> # The same way can the XLNetLMHeadModel be used to be trained by standard auto-regressive language modeling.
+            >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is very <mask>", add_special_tokens=False)).unsqueeze(0)  # We will predict the masked token
+            >>> labels = torch.tensor(tokenizer.encode("cute", add_special_tokens=False)).unsqueeze(0)
+            >>> assert labels.shape[0] == 1, 'only one word will be predicted'
+            >>> perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float)
+            >>> perm_mask[:, :, -1] = 1.0  # Previous tokens don't see last token as is done in standard auto-regressive lm training
+            >>> target_mapping = torch.zeros((1, 1, input_ids.shape[1]), dtype=torch.float)  # Shape [1, 1, seq_length] => let's predict one token
+            >>> target_mapping[0, 0, -1] = 1.0  # Our first (and only) prediction will be the last token of the sequence (the masked token)
 
-            outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping, labels=labels)
-            loss = outputs.loss
-            next_token_logits = outputs.logits  # Logits have shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size]
+            >>> outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping, labels=labels)
+            >>> loss = outputs.loss
+            >>> next_token_logits = outputs.logits  # Logits have shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size]
         """
         return_dict = return_dict if return_dict is not None else self.config.use_return_dict
         use_cache = self.training or (use_cache if use_cache is not None else self.config.use_cache)

src/transformers/pipelines.py

@@ -965,7 +965,7 @@ class TextClassificationPipeline(Pipeline):
 
         Args:
             args (:obj:`str` or :obj:`List[str]`):
-                One or several textts (or one list of prompts) to classify.
+                One or several texts (or one list of prompts) to classify.
 
         Return:
             A list or a list of list of :obj:`dict`: Each result comes as list of dictionaries with the
@@ -2661,18 +2661,18 @@ def pipeline(
 
     Examples::
 
-        from transformers import pipeline, AutoModelForTokenClassification, AutoTokenizer
+        >>> from transformers import pipeline, AutoModelForTokenClassification, AutoTokenizer
 
-        # Sentiment analysis pipeline
-        pipeline('sentiment-analysis')
+        >>> # Sentiment analysis pipeline
+        >>> pipeline('sentiment-analysis')
 
-        # Question answering pipeline, specifying the checkpoint identifier
-        pipeline('question-answering', model='distilbert-base-cased-distilled-squad', tokenizer='bert-base-cased')
+        >>> # Question answering pipeline, specifying the checkpoint identifier
+        >>> pipeline('question-answering', model='distilbert-base-cased-distilled-squad', tokenizer='bert-base-cased')
 
-        # Named entity recognition pipeline, passing in a specific model and tokenizer
-        model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
-        tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-        pipeline('ner', model=model, tokenizer=tokenizer)
+        >>> # Named entity recognition pipeline, passing in a specific model and tokenizer
+        >>> model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
+        >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
+        >>> pipeline('ner', model=model, tokenizer=tokenizer)
     """
     # Retrieve the task
     if task not in SUPPORTED_TASKS:

src/transformers/testing_utils.py

@@ -297,15 +297,15 @@ class CaptureLogger:
 
     Example:
 
-    from transformers import logging
-    from transformers.testing_utils import CaptureLogger
+    >>> from transformers import logging
+    >>> from transformers.testing_utils import CaptureLogger
 
-    msg = "Testing 1, 2, 3"
-    logging.set_verbosity_info()
-    logger = logging.get_logger("transformers.tokenization_bart")
-    with CaptureLogger(logger) as cl:
-        logger.info(msg)
-    assert cl.out, msg+"\n"
+    >>> msg = "Testing 1, 2, 3"
+    >>> logging.set_verbosity_info()
+    >>> logger = logging.get_logger("transformers.tokenization_bart")
+    >>> with CaptureLogger(logger) as cl:
+    ...     logger.info(msg)
+    >>> assert cl.out, msg+"\n"
     """
 
     def __init__(self, logger):

src/transformers/tokenization_auto.py

@@ -29,6 +29,7 @@ from .configuration_auto import (
     ElectraConfig,
     EncoderDecoderConfig,
     FlaubertConfig,
+    FSMTConfig,
     FunnelConfig,
     GPT2Config,
     LongformerConfig,
@@ -54,11 +55,13 @@ from .tokenization_bart import BartTokenizer, BartTokenizerFast
 from .tokenization_bert import BertTokenizer, BertTokenizerFast
 from .tokenization_bert_generation import BertGenerationTokenizer
 from .tokenization_bert_japanese import BertJapaneseTokenizer
+from .tokenization_bertweet import BertweetTokenizer
 from .tokenization_camembert import CamembertTokenizer
 from .tokenization_ctrl import CTRLTokenizer
 from .tokenization_distilbert import DistilBertTokenizer, DistilBertTokenizerFast
 from .tokenization_electra import ElectraTokenizer, ElectraTokenizerFast
 from .tokenization_flaubert import FlaubertTokenizer
+from .tokenization_fsmt import FSMTTokenizer
 from .tokenization_funnel import FunnelTokenizer, FunnelTokenizerFast
 from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast
 from .tokenization_longformer import LongformerTokenizer, LongformerTokenizerFast
@@ -68,6 +71,7 @@ from .tokenization_mbart import MBartTokenizer
 from .tokenization_mobilebert import MobileBertTokenizer, MobileBertTokenizerFast
 from .tokenization_openai import OpenAIGPTTokenizer, OpenAIGPTTokenizerFast
 from .tokenization_pegasus import PegasusTokenizer
+from .tokenization_phobert import PhobertTokenizer
 from .tokenization_reformer import ReformerTokenizer
 from .tokenization_retribert import RetriBertTokenizer, RetriBertTokenizerFast
 from .tokenization_roberta import RobertaTokenizer, RobertaTokenizerFast
@@ -96,6 +100,8 @@ TOKENIZER_MAPPING = OrderedDict(
         (MarianConfig, (MarianTokenizer, None)),
         (BartConfig, (BartTokenizer, BartTokenizerFast)),
         (LongformerConfig, (LongformerTokenizer, LongformerTokenizerFast)),
+        (RobertaConfig, (BertweetTokenizer, None)),
+        (RobertaConfig, (PhobertTokenizer, None)),
         (RobertaConfig, (RobertaTokenizer, RobertaTokenizerFast)),
         (ReformerConfig, (ReformerTokenizer, None)),
         (ElectraConfig, (ElectraTokenizer, ElectraTokenizerFast)),
@@ -109,6 +115,7 @@ TOKENIZER_MAPPING = OrderedDict(
         (FlaubertConfig, (FlaubertTokenizer, None)),
         (XLMConfig, (XLMTokenizer, None)),
         (CTRLConfig, (CTRLTokenizer, None)),
+        (FSMTConfig, (FSMTTokenizer, None)),
         (BertGenerationConfig, (BertGenerationTokenizer, None)),
     ]
 )
@@ -182,16 +189,16 @@ class AutoTokenizer:
 
         Examples::
 
-            from transformers import AutoTokenizer
+            >>> from transformers import AutoTokenizer
 
-            # Download vocabulary from S3 and cache.
-            tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
+            >>> # Download vocabulary from S3 and cache.
+            >>> tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
 
-            # Download vocabulary from S3 (user-uploaded) and cache.
-            tokenizer = AutoTokenizer.from_pretrained('dbmdz/bert-base-german-cased')
+            >>> # Download vocabulary from S3 (user-uploaded) and cache.
+            >>> tokenizer = AutoTokenizer.from_pretrained('dbmdz/bert-base-german-cased')
 
-            # If vocabulary files are in a directory (e.g. tokenizer was saved using `save_pretrained('./test/saved_model/')`)
-            tokenizer = AutoTokenizer.from_pretrained('./test/bert_saved_model/')
+            >>> # If vocabulary files are in a directory (e.g. tokenizer was saved using `save_pretrained('./test/saved_model/')`)
+            >>> tokenizer = AutoTokenizer.from_pretrained('./test/bert_saved_model/')
 
         """
         config = kwargs.pop("config", None)

src/transformers/tokenization_bertweet.py

@@ -0,0 +1,783 @@
+# coding=utf-8
+# Copyright (c) 2020, VinAI Research and the HuggingFace Inc. team.
+# Copyright 2018 The Open AI Team Authors and The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Tokenization classes for BERTweet """
+
+
+import html
+import os
+import re
+from shutil import copyfile
+from typing import List, Optional
+
+import regex
+
+from .tokenization_utils import PreTrainedTokenizer
+from .utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+VOCAB_FILES_NAMES = {
+    "vocab_file": "vocab.txt",
+    "merges_file": "bpe.codes",
+}
+
+PRETRAINED_VOCAB_FILES_MAP = {
+    "vocab_file": {
+        "vinai/bertweet-base": "https://s3.amazonaws.com/models.huggingface.co/bert/vinai/bertweet-base/vocab.txt",
+    },
+    "merges_file": {
+        "vinai/bertweet-base": "https://s3.amazonaws.com/models.huggingface.co/bert/vinai/bertweet-base/bpe.codes",
+    },
+}
+
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
+    "vinai/bertweet-base": 128,
+}
+
+
+def get_pairs(word):
+    """Return set of symbol pairs in a word.
+
+    Word is represented as tuple of symbols (symbols being variable-length strings).
+    """
+    pairs = set()
+    prev_char = word[0]
+    for char in word[1:]:
+        pairs.add((prev_char, char))
+        prev_char = char
+
+    pairs = set(pairs)
+    return pairs
+
+
+class BertweetTokenizer(PreTrainedTokenizer):
+    """
+    Constructs a BERTweet tokenizer. Peculiarities:
+
+    - Byte-Pair-Encoding
+
+    This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the methods. Users
+    should refer to the superclass for more information regarding methods.
+
+    Args:
+        vocab_file (:obj:`str`):
+            Path to the vocabulary file.
+        merges_file (:obj:`str`):
+            Path to the merges file.
+        normalization (:obj:`boolean`, defaults to False)
+            Whether to apply a normalization pre-process.
+        bos_token (:obj:`string`, `optional`, defaults to "<s>"):
+            The beginning of sequence token that was used during pre-training. Can be used a sequence classifier token.
+
+            .. note::
+
+                When building a sequence using special tokens, this is not the token that is used for the beginning
+                of sequence. The token used is the :obj:`cls_token`.
+        eos_token (:obj:`string`, `optional`, defaults to "</s>"):
+            The end of sequence token.
+
+            .. note::
+
+                When building a sequence using special tokens, this is not the token that is used for the end
+                of sequence. The token used is the :obj:`sep_token`.
+        sep_token (:obj:`string`, `optional`, defaults to "</s>"):
+            The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences
+            for sequence classification or for a text and a question for question answering.
+            It is also used as the last token of a sequence built with special tokens.
+        cls_token (:obj:`string`, `optional`, defaults to "<s>"):
+            The classifier token which is used when doing sequence classification (classification of the whole
+            sequence instead of per-token classification). It is the first token of the sequence when built with
+            special tokens.
+        unk_token (:obj:`string`, `optional`, defaults to "<unk>"):
+            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
+            token instead.
+        pad_token (:obj:`string`, `optional`, defaults to "<pad>"):
+            The token used for padding, for example when batching sequences of different lengths.
+        mask_token (:obj:`string`, `optional`, defaults to "<mask>"):
+            The token used for masking values. This is the token used when training this model with masked language
+            modeling. This is the token which the model will try to predict.
+    """
+
+    vocab_files_names = VOCAB_FILES_NAMES
+    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
+    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+
+    def __init__(
+        self,
+        vocab_file,
+        merges_file,
+        normalization=False,
+        bos_token="<s>",
+        eos_token="</s>",
+        sep_token="</s>",
+        cls_token="<s>",
+        unk_token="<unk>",
+        pad_token="<pad>",
+        mask_token="<mask>",
+        **kwargs
+    ):
+        super().__init__(
+            max_len=128,
+            bos_token=bos_token,
+            eos_token=eos_token,
+            unk_token=unk_token,
+            sep_token=sep_token,
+            cls_token=cls_token,
+            pad_token=pad_token,
+            mask_token=mask_token,
+            **kwargs,
+        )
+
+        try:
+            from emoji import demojize
+
+            self.demojizer = demojize
+        except ImportError:
+            logger.warning(
+                "emoji is not installed, thus not converting emoticons or emojis into text. Please install emoji: pip3 install emoji"
+            )
+            self.demojizer = None
+
+        self.vocab_file = vocab_file
+        self.merges_file = merges_file
+
+        self.encoder = {}
+        self.encoder[self.bos_token] = 0
+        self.encoder[self.pad_token] = 1
+        self.encoder[self.eos_token] = 2
+        self.encoder[self.unk_token] = 3
+
+        self.add_from_file(vocab_file)
+
+        self.decoder = {v: k for k, v in self.encoder.items()}
+
+        with open(merges_file, encoding="utf-8") as merges_handle:
+            merges = merges_handle.read().split("\n")[:-1]
+        merges = [tuple(merge.split()[:-1]) for merge in merges]
+        self.bpe_ranks = dict(zip(merges, range(len(merges))))
+        self.cache = {}
+
+        self.normalization = normalization
+        self.tweetPreprocessor = TweetTokenizer()
+
+        self.special_puncts = {"’": "'", "…": "..."}
+
+    def build_inputs_with_special_tokens(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Build model inputs from a sequence or a pair of sequence for sequence classification tasks
+        by concatenating and adding special tokens.
+        A BERTweet sequence has the following format:
+
+        - single sequence: ``<s> X </s>``
+        - pair of sequences: ``<s> A </s></s> B </s>``
+
+        Args:
+            token_ids_0 (:obj:`List[int]`):
+                List of IDs to which the special tokens will be added
+            token_ids_1 (:obj:`List[int]`, `optional`, defaults to :obj:`None`):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            :obj:`List[int]`: list of `input IDs <../glossary.html#input-ids>`__ with the appropriate special tokens.
+        """
+
+        if token_ids_1 is None:
+            return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
+        cls = [self.cls_token_id]
+        sep = [self.sep_token_id]
+        return cls + token_ids_0 + sep + sep + token_ids_1 + sep
+
+    def get_special_tokens_mask(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
+    ) -> List[int]:
+        """
+        Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding
+        special tokens using the tokenizer ``prepare_for_model`` methods.
+
+        Args:
+            token_ids_0 (:obj:`List[int]`):
+                List of ids.
+            token_ids_1 (:obj:`List[int]`, `optional`, defaults to :obj:`None`):
+                Optional second list of IDs for sequence pairs.
+            already_has_special_tokens (:obj:`bool`, `optional`, defaults to :obj:`False`):
+                Set to True if the token list is already formatted with special tokens for the model
+
+        Returns:
+            :obj:`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
+        """
+
+        if already_has_special_tokens:
+            if token_ids_1 is not None:
+                raise ValueError(
+                    "You should not supply a second sequence if the provided sequence of "
+                    "ids is already formated with special tokens for the model."
+                )
+            return list(map(lambda x: 1 if x in [self.sep_token_id, self.cls_token_id] else 0, token_ids_0))
+
+        if token_ids_1 is None:
+            return [1] + ([0] * len(token_ids_0)) + [1]
+        return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]
+
+    def create_token_type_ids_from_sequences(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Creates a mask from the two sequences passed to be used in a sequence-pair classification task.
+        BERTweet does not make use of token type ids, therefore a list of zeros is returned.
+
+        Args:
+            token_ids_0 (:obj:`List[int]`):
+                List of ids.
+            token_ids_1 (:obj:`List[int]`, `optional`, defaults to :obj:`None`):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            :obj:`List[int]`: List of zeros.
+
+        """
+
+        sep = [self.sep_token_id]
+        cls = [self.cls_token_id]
+
+        if token_ids_1 is None:
+            return len(cls + token_ids_0 + sep) * [0]
+        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]
+
+    @property
+    def vocab_size(self):
+        return len(self.encoder)
+
+    def get_vocab(self):
+        return dict(self.encoder, **self.added_tokens_encoder)
+
+    def bpe(self, token):
+        if token in self.cache:
+            return self.cache[token]
+        word = tuple(token)
+        word = tuple(list(word[:-1]) + [word[-1] + "</w>"])
+        pairs = get_pairs(word)
+
+        if not pairs:
+            return token
+
+        while True:
+            bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf")))
+            if bigram not in self.bpe_ranks:
+                break
+            first, second = bigram
+            new_word = []
+            i = 0
+            while i < len(word):
+                try:
+                    j = word.index(first, i)
+                except ValueError:
+                    new_word.extend(word[i:])
+                    break
+                else:
+                    new_word.extend(word[i:j])
+                    i = j
+
+                if word[i] == first and i < len(word) - 1 and word[i + 1] == second:
+                    new_word.append(first + second)
+                    i += 2
+                else:
+                    new_word.append(word[i])
+                    i += 1
+            new_word = tuple(new_word)
+            word = new_word
+            if len(word) == 1:
+                break
+            else:
+                pairs = get_pairs(word)
+        word = "@@ ".join(word)
+        word = word[:-4]
+        self.cache[token] = word
+        return word
+
+    def _tokenize(self, text):
+        """Tokenize a string."""
+        if self.normalization:  # Perform Tweet normalization before performing BPE
+            text = self.normalizeTweet(text)
+
+        split_tokens = []
+        words = re.findall(r"\S+\n?", text)
+        for token in words:
+            split_tokens.extend([t for t in self.bpe(token).split(" ")])
+        return split_tokens
+
+    def normalizeTweet(self, tweet):
+        """
+        Normalize a raw Tweet
+        """
+        for punct in self.special_puncts:
+            tweet = tweet.replace(punct, self.special_puncts[punct])
+
+        tokens = self.tweetPreprocessor.tokenize(tweet)
+        normTweet = " ".join([self.normalizeToken(token) for token in tokens])
+
+        normTweet = (
+            normTweet.replace("cannot ", "can not ")
+            .replace("n't ", " n't ")
+            .replace("n 't ", " n't ")
+            .replace("ca n't", "can't")
+            .replace("ai n't", "ain't")
+        )
+        normTweet = (
+            normTweet.replace("'m ", " 'm ")
+            .replace("'re ", " 're ")
+            .replace("'s ", " 's ")
+            .replace("'ll ", " 'll ")
+            .replace("'d ", " 'd ")
+            .replace("'ve ", " 've ")
+        )
+        normTweet = (
+            normTweet.replace(" p . m .", "  p.m.")
+            .replace(" p . m ", " p.m ")
+            .replace(" a . m .", " a.m.")
+            .replace(" a . m ", " a.m ")
+        )
+
+        return " ".join(normTweet.split())
+
+    def normalizeToken(self, token):
+        """
+        Normalize tokens in a Tweet
+        """
+        lowercased_token = token.lower()
+        if token.startswith("@"):
+            return "@USER"
+        elif lowercased_token.startswith("http") or lowercased_token.startswith("www"):
+            return "HTTPURL"
+        elif len(token) == 1:
+            if token in self.special_puncts:
+                return self.special_puncts[token]
+            if self.demojizer is not None:
+                return self.demojizer(token)
+            else:
+                return token
+        else:
+            return token
+
+    def _convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        return self.encoder.get(token, self.encoder.get(self.unk_token))
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        return self.decoder.get(index, self.unk_token)
+
+    def convert_tokens_to_string(self, tokens):
+        """ Converts a sequence of tokens (string) in a single string. """
+        out_string = " ".join(tokens).replace("@@ ", "").strip()
+        return out_string
+
+    def save_vocabulary(self, save_directory):
+        """
+        Save the vocabulary and special tokens file to a directory.
+        Args:
+            save_directory (:obj:`str`):
+                The directory in which to save the vocabulary.
+        Returns:
+            :obj:`Tuple(str)`: Paths to the files saved.
+        """
+        if not os.path.isdir(save_directory):
+            logger.error("Vocabulary path ({}) should be a directory".format(save_directory))
+            return
+        out_vocab_file = os.path.join(save_directory, VOCAB_FILES_NAMES["vocab_file"])
+        out_merge_file = os.path.join(save_directory, VOCAB_FILES_NAMES["merges_file"])
+
+        if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
+            copyfile(self.vocab_file, out_vocab_file)
+
+        if os.path.abspath(self.merges_file) != os.path.abspath(out_merge_file):
+            copyfile(self.merges_file, out_merge_file)
+
+        return out_vocab_file, out_merge_file
+
+    # def decode(self, token_ids, skip_special_tokens=False, clean_up_tokenization_spaces=True):
+    #     filtered_tokens = ' '.join(self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens))
+    #     tokens_generated_so_far = re.sub('(@@ )', '', string=filtered_tokens)
+    #     tokens_generated_so_far = re.sub('(@@ ?$)', '', string=tokens_generated_so_far)
+    #     return ''.join(tokens_generated_so_far)
+
+    def add_from_file(self, f):
+        """
+        Loads a pre-existing dictionary from a text file and adds its symbols
+        to this instance.
+        """
+        if isinstance(f, str):
+            try:
+                with open(f, "r", encoding="utf-8") as fd:
+                    self.add_from_file(fd)
+            except FileNotFoundError as fnfe:
+                raise fnfe
+            except UnicodeError:
+                raise Exception("Incorrect encoding detected in {}, please " "rebuild the dataset".format(f))
+            return
+
+        lines = f.readlines()
+        for lineTmp in lines:
+            line = lineTmp.strip()
+            idx = line.rfind(" ")
+            if idx == -1:
+                raise ValueError("Incorrect dictionary format, expected '<token> <cnt>'")
+            word = line[:idx]
+            self.encoder[word] = len(self.encoder)
+
+
+# Natural Language Toolkit: Twitter Tokenizer
+#
+# Copyright (C) 2001-2020 NLTK Project
+# Author: Christopher Potts <cgpotts@stanford.edu>
+#         Ewan Klein <ewan@inf.ed.ac.uk> (modifications)
+#         Pierpaolo Pantone <> (modifications)
+# URL: <http://nltk.org/>
+# For license information, see LICENSE.TXT
+#
+
+
+"""
+Twitter-aware tokenizer, designed to be flexible and easy to adapt to new
+domains and tasks. The basic logic is this:
+
+1. The tuple regex_strings defines a list of regular expression
+   strings.
+
+2. The regex_strings strings are put, in order, into a compiled
+   regular expression object called word_re.
+
+3. The tokenization is done by word_re.findall(s), where s is the
+   user-supplied string, inside the tokenize() method of the class
+   Tokenizer.
+
+4. When instantiating Tokenizer objects, there is a single option:
+   preserve_case.  By default, it is set to True. If it is set to
+   False, then the tokenizer will downcase everything except for
+   emoticons.
+
+"""
+
+
+######################################################################
+#
+# import regex  # https://github.com/nltk/nltk/issues/2409
+# import html
+#
+######################################################################
+# The following strings are components in the regular expression
+# that is used for tokenizing. It's important that phone_number
+# appears first in the final regex (since it can contain whitespace).
+# It also could matter that tags comes after emoticons, due to the
+# possibility of having text like
+#
+#     <:| and some text >:)
+#
+# Most importantly, the final element should always be last, since it
+# does a last ditch whitespace-based tokenization of whatever is left.
+
+# ToDo: Update with http://en.wikipedia.org/wiki/List_of_emoticons ?
+
+# This particular element is used in a couple ways, so we define it
+# with a name:
+EMOTICONS = r"""
+    (?:
+      [<>]?
+      [:;=8]                     # eyes
+      [\-o\*\']?                 # optional nose
+      [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
+      |
+      [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
+      [\-o\*\']?                 # optional nose
+      [:;=8]                     # eyes
+      [<>]?
+      |
+      <3                         # heart
+    )"""
+
+# URL pattern due to John Gruber, modified by Tom Winzig. See
+# https://gist.github.com/winzig/8894715
+
+URLS = r"""			# Capture 1: entire matched URL
+  (?:
+  https?:				# URL protocol and colon
+    (?:
+      /{1,3}				# 1-3 slashes
+      |					#   or
+      [a-z0-9%]				# Single letter or digit or '%'
+                                       # (Trying not to match e.g. "URI::Escape")
+    )
+    |					#   or
+                                       # looks like domain name followed by a slash:
+    [a-z0-9.\-]+[.]
+    (?:[a-z]{2,13})
+    /
+  )
+  (?:					# One or more:
+    [^\s()<>{}\[\]]+			# Run of non-space, non-()<>{}[]
+    |					#   or
+    \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...)
+    |
+    \([^\s]+?\)				# balanced parens, non-recursive: (...)
+  )+
+  (?:					# End with:
+    \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...)
+    |
+    \([^\s]+?\)				# balanced parens, non-recursive: (...)
+    |					#   or
+    [^\s`!()\[\]{};:'".,<>?«»“”‘’]	# not a space or one of these punct chars
+  )
+  |					# OR, the following to match naked domains:
+  (?:
+    (?<!@)			        # not preceded by a @, avoid matching foo@_gmail.com_
+    [a-z0-9]+
+    (?:[.\-][a-z0-9]+)*
+    [.]
+    (?:[a-z]{2,13})
+    \b
+    /?
+    (?!@)			        # not succeeded by a @,
+                            # avoid matching "foo.na" in "foo.na@example.com"
+  )
+"""
+
+# The components of the tokenizer:
+REGEXPS = (
+    URLS,
+    # Phone numbers:
+    r"""
+    (?:
+      (?:            # (international)
+        \+?[01]
+        [ *\-.\)]*
+      )?
+      (?:            # (area code)
+        [\(]?
+        \d{3}
+        [ *\-.\)]*
+      )?
+      \d{3}          # exchange
+      [ *\-.\)]*
+      \d{4}          # base
+    )""",
+    # ASCII Emoticons
+    EMOTICONS,
+    # HTML tags:
+    r"""<[^>\s]+>""",
+    # ASCII Arrows
+    r"""[\-]+>|<[\-]+""",
+    # Twitter username:
+    r"""(?:@[\w_]+)""",
+    # Twitter hashtags:
+    r"""(?:\#+[\w_]+[\w\'_\-]*[\w_]+)""",
+    # email addresses
+    r"""[\w.+-]+@[\w-]+\.(?:[\w-]\.?)+[\w-]""",
+    # Remaining word types:
+    r"""
+    (?:[^\W\d_](?:[^\W\d_]|['\-_])+[^\W\d_]) # Words with apostrophes or dashes.
+    |
+    (?:[+\-]?\d+[,/.:-]\d+[+\-]?)  # Numbers, including fractions, decimals.
+    |
+    (?:[\w_]+)                     # Words without apostrophes or dashes.
+    |
+    (?:\.(?:\s*\.){1,})            # Ellipsis dots.
+    |
+    (?:\S)                         # Everything else that isn't whitespace.
+    """,
+)
+
+######################################################################
+# This is the core tokenizing regex:
+
+WORD_RE = regex.compile(r"""(%s)""" % "|".join(REGEXPS), regex.VERBOSE | regex.I | regex.UNICODE)
+
+# WORD_RE performs poorly on these patterns:
+HANG_RE = regex.compile(r"([^a-zA-Z0-9])\1{3,}")
+
+# The emoticon string gets its own regex so that we can preserve case for
+# them as needed:
+EMOTICON_RE = regex.compile(EMOTICONS, regex.VERBOSE | regex.I | regex.UNICODE)
+
+# These are for regularizing HTML entities to Unicode:
+ENT_RE = regex.compile(r"&(#?(x?))([^&;\s]+);")
+
+
+######################################################################
+# Functions for converting html entities
+######################################################################
+
+
+def _str_to_unicode(text, encoding=None, errors="strict"):
+    if encoding is None:
+        encoding = "utf-8"
+    if isinstance(text, bytes):
+        return text.decode(encoding, errors)
+    return text
+
+
+def _replace_html_entities(text, keep=(), remove_illegal=True, encoding="utf-8"):
+    """
+    Remove entities from text by converting them to their
+    corresponding unicode character.
+
+    :param text: a unicode string or a byte string encoded in the given
+    `encoding` (which defaults to 'utf-8').
+
+    :param list keep:  list of entity names which should not be replaced.\
+    This supports both numeric entities (``&#nnnn;`` and ``&#hhhh;``)
+    and named entities (such as ``&nbsp;`` or ``&gt;``).
+
+    :param bool remove_illegal: If `True`, entities that can't be converted are\
+    removed. Otherwise, entities that can't be converted are kept "as
+    is".
+
+    :returns: A unicode string with the entities removed.
+
+    See https://github.com/scrapy/w3lib/blob/master/w3lib/html.py
+
+        >>> from nltk.tokenize.casual import _replace_html_entities
+        >>> _replace_html_entities(b'Price: &pound;100')
+        'Price: \\xa3100'
+        >>> print(_replace_html_entities(b'Price: &pound;100'))
+        Price: £100
+        >>>
+    """
+
+    def _convert_entity(match):
+        entity_body = match.group(3)
+        if match.group(1):
+            try:
+                if match.group(2):
+                    number = int(entity_body, 16)
+                else:
+                    number = int(entity_body, 10)
+                # Numeric character references in the 80-9F range are typically
+                # interpreted by browsers as representing the characters mapped
+                # to bytes 80-9F in the Windows-1252 encoding. For more info
+                # see: https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Similar_character_sets
+                if 0x80 <= number <= 0x9F:
+                    return bytes((number,)).decode("cp1252")
+            except ValueError:
+                number = None
+        else:
+            if entity_body in keep:
+                return match.group(0)
+            else:
+                number = html.entities.name2codepoint.get(entity_body)
+        if number is not None:
+            try:
+                return chr(number)
+            except (ValueError, OverflowError):
+                pass
+
+        return "" if remove_illegal else match.group(0)
+
+    return ENT_RE.sub(_convert_entity, _str_to_unicode(text, encoding))
+
+
+######################################################################
+
+
+class TweetTokenizer:
+    r"""
+    Tokenizer for tweets.
+
+        >>> from nltk.tokenize import TweetTokenizer
+        >>> tknzr = TweetTokenizer()
+        >>> s0 = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--"
+        >>> tknzr.tokenize(s0)
+        ['This', 'is', 'a', 'cooool', '#dummysmiley', ':', ':-)', ':-P', '<3', 'and', 'some', 'arrows', '<', '>', '->', '<--']
+
+    Examples using `strip_handles` and `reduce_len parameters`:
+
+        >>> tknzr = TweetTokenizer(strip_handles=True, reduce_len=True)
+        >>> s1 = '@remy: This is waaaaayyyy too much for you!!!!!!'
+        >>> tknzr.tokenize(s1)
+        [':', 'This', 'is', 'waaayyy', 'too', 'much', 'for', 'you', '!', '!', '!']
+    """
+
+    def __init__(self, preserve_case=True, reduce_len=False, strip_handles=False):
+        self.preserve_case = preserve_case
+        self.reduce_len = reduce_len
+        self.strip_handles = strip_handles
+
+    def tokenize(self, text):
+        """
+        :param text: str
+        :rtype: list(str)
+        :return: a tokenized list of strings; concatenating this list returns\
+        the original string if `preserve_case=False`
+        """
+        # Fix HTML character entities:
+        text = _replace_html_entities(text)
+        # Remove username handles
+        if self.strip_handles:
+            text = remove_handles(text)
+        # Normalize word lengthening
+        if self.reduce_len:
+            text = reduce_lengthening(text)
+        # Shorten problematic sequences of characters
+        safe_text = HANG_RE.sub(r"\1\1\1", text)
+        # Tokenize:
+        words = WORD_RE.findall(safe_text)
+        # Possibly alter the case, but avoid changing emoticons like :D into :d:
+        if not self.preserve_case:
+            words = list(map((lambda x: x if EMOTICON_RE.search(x) else x.lower()), words))
+        return words
+
+
+######################################################################
+# Normalization Functions
+######################################################################
+
+
+def reduce_lengthening(text):
+    """
+    Replace repeated character sequences of length 3 or greater with sequences
+    of length 3.
+    """
+    pattern = regex.compile(r"(.)\1{2,}")
+    return pattern.sub(r"\1\1\1", text)
+
+
+def remove_handles(text):
+    """
+    Remove Twitter username handles from text.
+    """
+    pattern = regex.compile(
+        r"(?<![A-Za-z0-9_!@#\$%&*])@(([A-Za-z0-9_]){20}(?!@))|(?<![A-Za-z0-9_!@#\$%&*])@(([A-Za-z0-9_]){1,19})(?![A-Za-z0-9_]*@)"
+    )
+    # Substitute handles with ' ' to ensure that text on either side of removed handles are tokenized correctly
+    return pattern.sub(" ", text)
+
+
+######################################################################
+# Tokenization Function
+######################################################################
+
+
+def casual_tokenize(text, preserve_case=True, reduce_len=False, strip_handles=False):
+    """
+    Convenience function for wrapping the tokenizer.
+    """
+    return TweetTokenizer(preserve_case=preserve_case, reduce_len=reduce_len, strip_handles=strip_handles).tokenize(
+        text
+    )
+
+
+###############################################################################

src/transformers/tokenization_dpr.py

@@ -278,18 +278,18 @@ class CustomDPRReaderTokenizerMixin:
 
         Examples::
 
-            from transformers import DPRReader, DPRReaderTokenizer
-            tokenizer = DPRReaderTokenizer.from_pretrained('facebook/dpr-reader-single-nq-base')
-            model = DPRReader.from_pretrained('facebook/dpr-reader-single-nq-base')
-            encoded_inputs = tokenizer(
-                    questions=["What is love ?"],
-                    titles=["Haddaway"],
-                    texts=["'What Is Love' is a song recorded by the artist Haddaway"],
-                    return_tensors='pt'
-                )
-            outputs = model(**encoded_inputs)
-            predicted_spans = tokenizer.decode_best_spans(encoded_inputs, outputs)
-            print(predicted_spans[0].text)  # best span
+            >>> from transformers import DPRReader, DPRReaderTokenizer
+            >>> tokenizer = DPRReaderTokenizer.from_pretrained('facebook/dpr-reader-single-nq-base')
+            >>> model = DPRReader.from_pretrained('facebook/dpr-reader-single-nq-base')
+            >>> encoded_inputs = tokenizer(
+            ...         questions=["What is love ?"],
+            ...         titles=["Haddaway"],
+            ...         texts=["'What Is Love' is a song recorded by the artist Haddaway"],
+            ...         return_tensors='pt'
+            ...     )
+            >>> outputs = model(**encoded_inputs)
+            >>> predicted_spans = tokenizer.decode_best_spans(encoded_inputs, outputs)
+            >>> print(predicted_spans[0].text)  # best span
 
         """
         input_ids = reader_input["input_ids"]

src/transformers/tokenization_fsmt.py

@@ -0,0 +1,535 @@
+# coding=utf-8
+# Copyright 2019 The Open AI Team Authors and The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tokenization classes for FSMT."""
+
+
+import json
+import logging
+import os
+import re
+import unicodedata
+from typing import Dict, List, Optional
+
+import sacremoses as sm
+
+from .file_utils import add_start_docstrings
+from .tokenization_utils import BatchEncoding, PreTrainedTokenizer
+from .tokenization_utils_base import PREPARE_SEQ2SEQ_BATCH_DOCSTRING
+
+
+logger = logging.getLogger(__name__)
+
+VOCAB_FILES_NAMES = {
+    "src_vocab_file": "vocab-src.json",
+    "tgt_vocab_file": "vocab-tgt.json",
+    "merges_file": "merges.txt",
+}
+
+PRETRAINED_VOCAB_FILES_MAP = {}
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {}
+PRETRAINED_INIT_CONFIGURATION = {}
+
+
+def get_pairs(word):
+    """
+    Return set of symbol pairs in a word.
+    word is represented as tuple of symbols (symbols being variable-length strings)
+    """
+    pairs = set()
+    prev_char = word[0]
+    for char in word[1:]:
+        pairs.add((prev_char, char))
+        prev_char = char
+    return pairs
+
+
+def replace_unicode_punct(text):
+    """
+    Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/replace-unicode-punctuation.perl
+    """
+    text = text.replace("，", ",")
+    text = re.sub(r"。\s*", ". ", text)
+    text = text.replace("、", ",")
+    text = text.replace("”", '"')
+    text = text.replace("“", '"')
+    text = text.replace("∶", ":")
+    text = text.replace("：", ":")
+    text = text.replace("？", "?")
+    text = text.replace("《", '"')
+    text = text.replace("》", '"')
+    text = text.replace("）", ")")
+    text = text.replace("！", "!")
+    text = text.replace("（", "(")
+    text = text.replace("；", ";")
+    text = text.replace("１", "1")
+    text = text.replace("」", '"')
+    text = text.replace("「", '"')
+    text = text.replace("０", "0")
+    text = text.replace("３", "3")
+    text = text.replace("２", "2")
+    text = text.replace("５", "5")
+    text = text.replace("６", "6")
+    text = text.replace("９", "9")
+    text = text.replace("７", "7")
+    text = text.replace("８", "8")
+    text = text.replace("４", "4")
+    text = re.sub(r"．\s*", ". ", text)
+    text = text.replace("～", "~")
+    text = text.replace("’", "'")
+    text = text.replace("…", "...")
+    text = text.replace("━", "-")
+    text = text.replace("〈", "<")
+    text = text.replace("〉", ">")
+    text = text.replace("【", "[")
+    text = text.replace("】", "]")
+    text = text.replace("％", "%")
+    return text
+
+
+def remove_non_printing_char(text):
+    """
+    Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/remove-non-printing-char.perl
+    """
+    output = []
+    for char in text:
+        cat = unicodedata.category(char)
+        if cat.startswith("C"):
+            continue
+        output.append(char)
+    return "".join(output)
+
+
+# Porting notes:
+# this one is modeled after XLMTokenizer
+#
+# added:
+# - src_vocab_file,
+# - tgt_vocab_file,
+# - langs,
+
+
+class FSMTTokenizer(PreTrainedTokenizer):
+    """
+    BPE tokenizer for FSMT (fairseq transformer)
+    See: https://github.com/pytorch/fairseq/tree/master/examples/wmt19
+
+    - Moses preprocessing & tokenization for most supported languages
+    - (optionally) lower case & normalize all inputs text
+    - argument ``special_tokens`` and function ``set_special_tokens``, can be used to add additional symbols \
+      (ex: "__classify__") to a vocabulary
+    - `langs` defines a pair of languages
+
+    This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the methods. Users
+    should refer to the superclass for more information regarding methods.
+
+    Args:
+        langs (:obj:`List[str]`):
+            a list of two languages to translate from and to, e.g. ``["en", "ru"]``.
+        src_vocab_file (:obj:`string`):
+            Source language vocabulary file.
+        tgt_vocab_file (:obj:`string`):
+            Target language vocabulary file.
+        merges_file (:obj:`string`):
+            Merges file.
+        do_lower_case (:obj:`bool`, `optional`, defaults to :obj:`True`):
+            Whether to lowercase the input when tokenizing.
+        unk_token (:obj:`string`, `optional`, defaults to "<unk>"):
+            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
+            token instead.
+        bos_token (:obj:`string`, `optional`, defaults to "<s>"):
+            The beginning of sequence token that was used during pre-training. Can be used a sequence classifier token.
+
+            .. note::
+
+                When building a sequence using special tokens, this is not the token that is used for the beginning
+                of sequence. The token used is the :obj:`cls_token`.
+        sep_token (:obj:`string`, `optional`, defaults to "</s>"):
+            The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences
+            for sequence classification or for a text and a question for question answering.
+            It is also used as the last token of a sequence built with special tokens.
+        pad_token (:obj:`string`, `optional`, defaults to "<pad>"):
+            The token used for padding, for example when batching sequences of different lengths.
+
+    """
+
+    vocab_files_names = VOCAB_FILES_NAMES
+    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
+    pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION
+    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+
+    def __init__(
+        self,
+        langs=None,
+        src_vocab_file=None,
+        tgt_vocab_file=None,
+        merges_file=None,
+        unk_token="<unk>",
+        bos_token="<s>",
+        sep_token="</s>",
+        pad_token="<pad>",
+        **kwargs
+    ):
+        super().__init__(
+            unk_token=unk_token,
+            bos_token=bos_token,
+            sep_token=sep_token,
+            pad_token=pad_token,
+            **kwargs,
+        )
+
+        self.src_vocab_file = src_vocab_file
+        self.tgt_vocab_file = tgt_vocab_file
+        self.merges_file = merges_file
+
+        # cache of sm.MosesPunctNormalizer instance
+        self.cache_moses_punct_normalizer = dict()
+        # cache of sm.MosesTokenizer instance
+        self.cache_moses_tokenizer = dict()
+        self.cache_moses_detokenizer = dict()
+
+        if langs and len(langs) == 2:
+            self.src_lang, self.tgt_lang = langs
+        else:
+            raise ValueError(
+                f"arg `langs` needs to be a list of 2 langs, e.g. ['en', 'ru'], but got {langs}. "
+                "Usually that means that tokenizer can't find a mapping for the given model path "
+                "in PRETRAINED_VOCAB_FILES_MAP, and other maps of this tokenizer."
+            )
+
+        with open(src_vocab_file, encoding="utf-8") as src_vocab_handle:
+            self.encoder = json.load(src_vocab_handle)
+        with open(tgt_vocab_file, encoding="utf-8") as tgt_vocab_handle:
+            tgt_vocab = json.load(tgt_vocab_handle)
+            self.decoder = {v: k for k, v in tgt_vocab.items()}
+        with open(merges_file, encoding="utf-8") as merges_handle:
+            merges = merges_handle.read().split("\n")[:-1]
+        merges = [tuple(merge.split()[:2]) for merge in merges]
+        self.bpe_ranks = dict(zip(merges, range(len(merges))))
+        self.cache = {}
+
+    # hack override
+    def get_vocab(self) -> Dict[str, int]:
+        return self.get_src_vocab()
+
+    # hack override
+    @property
+    def vocab_size(self) -> int:
+        return self.src_vocab_size
+
+    def moses_punct_norm(self, text, lang):
+        if lang not in self.cache_moses_punct_normalizer:
+            punct_normalizer = sm.MosesPunctNormalizer(lang=lang)
+            self.cache_moses_punct_normalizer[lang] = punct_normalizer
+        return self.cache_moses_punct_normalizer[lang].normalize(text)
+
+    def moses_tokenize(self, text, lang):
+        if lang not in self.cache_moses_tokenizer:
+            moses_tokenizer = sm.MosesTokenizer(lang=lang)
+            self.cache_moses_tokenizer[lang] = moses_tokenizer
+        return self.cache_moses_tokenizer[lang].tokenize(
+            text, aggressive_dash_splits=True, return_str=False, escape=True
+        )
+
+    def moses_detokenize(self, tokens, lang):
+        if lang not in self.cache_moses_tokenizer:
+            moses_detokenizer = sm.MosesDetokenizer(lang=self.tgt_lang)
+            self.cache_moses_detokenizer[lang] = moses_detokenizer
+        return self.cache_moses_detokenizer[lang].detokenize(tokens)
+
+    def moses_pipeline(self, text, lang):
+        text = replace_unicode_punct(text)
+        text = self.moses_punct_norm(text, lang)
+        text = remove_non_printing_char(text)
+        return text
+
+    @property
+    def src_vocab_size(self):
+        return len(self.encoder)
+
+    @property
+    def tgt_vocab_size(self):
+        return len(self.decoder)
+
+    def get_src_vocab(self):
+        return dict(self.encoder, **self.added_tokens_encoder)
+
+    def get_tgt_vocab(self):
+        return dict(self.decoder, **self.added_tokens_decoder)
+
+    def bpe(self, token):
+        word = tuple(token[:-1]) + (token[-1] + "</w>",)
+        if token in self.cache:
+            return self.cache[token]
+        pairs = get_pairs(word)
+
+        if not pairs:
+            return token + "</w>"
+
+        while True:
+            bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf")))
+            if bigram not in self.bpe_ranks:
+                break
+            first, second = bigram
+            new_word = []
+            i = 0
+            while i < len(word):
+                try:
+                    j = word.index(first, i)
+                except ValueError:
+                    new_word.extend(word[i:])
+                    break
+                else:
+                    new_word.extend(word[i:j])
+                    i = j
+
+                if word[i] == first and i < len(word) - 1 and word[i + 1] == second:
+                    new_word.append(first + second)
+                    i += 2
+                else:
+                    new_word.append(word[i])
+                    i += 1
+            new_word = tuple(new_word)
+            word = new_word
+            if len(word) == 1:
+                break
+            else:
+                pairs = get_pairs(word)
+        word = " ".join(word)
+        if word == "\n  </w>":
+            word = "\n</w>"
+        self.cache[token] = word
+        return word
+
+    def _tokenize(self, text, lang="en", bypass_tokenizer=False):
+        """
+        Tokenize a string given language code using Moses.
+
+        Details of tokenization:
+        - [sacremoses](https://github.com/alvations/sacremoses): port of Moses
+            - Install with `pip install sacremoses`
+
+        Args:
+            - lang: ISO language code (default = 'en') (string). Languages should belong of the model supported languages. However, we don't enforce it.
+            - bypass_tokenizer: Allow users to preprocess and tokenize the sentences externally (default = False) (bool). If True, we only apply BPE.
+
+        Returns:
+            List of tokens.
+        """
+        # ignore `lang` which is currently isn't explicitly passed in tokenization_utils.py and always results in lang=en
+        # if lang != self.src_lang:
+        #     raise ValueError(f"Expected lang={self.src_lang}, but got {lang}")
+        lang = self.src_lang
+
+        if bypass_tokenizer:
+            text = text.split()
+        else:
+            text = self.moses_pipeline(text, lang=lang)
+            text = self.moses_tokenize(text, lang=lang)
+
+        split_tokens = []
+        for token in text:
+            if token:
+                split_tokens.extend([t for t in self.bpe(token).split(" ")])
+
+        return split_tokens
+
+    def _convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        return self.encoder.get(token, self.encoder.get(self.unk_token))
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        return self.decoder.get(index, self.unk_token)
+
+    def convert_tokens_to_string(self, tokens):
+        """ Converts a sequence of tokens (string) in a single string. """
+
+        # remove BPE
+        tokens = [t.replace(" ", "").replace("</w>", " ") for t in tokens]
+        tokens = "".join(tokens).split()
+        # detokenize
+        text = self.moses_detokenize(tokens, self.tgt_lang)
+        return text
+
+    def build_inputs_with_special_tokens(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Build model inputs from a sequence or a pair of sequence for sequence classification tasks
+        by concatenating and adding special tokens.
+        A FAIRSEQ_TRANSFORMER sequence has the following format:
+
+        - single sequence: ``<s> X </s>``
+        - pair of sequences: ``<s> A </s> B </s>``
+
+        Args:
+            token_ids_0 (:obj:`List[int]`):
+                List of IDs to which the special tokens will be added
+            token_ids_1 (:obj:`List[int]`, `optional`):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            :obj:`List[int]`: list of `input IDs <../glossary.html#input-ids>`__ with the appropriate special tokens.
+
+        """
+        sep = [self.sep_token_id]
+
+        # no bos used in fairseq
+        if token_ids_1 is None:
+            return token_ids_0 + sep
+        return token_ids_0 + sep + token_ids_1 + sep
+
+    def get_special_tokens_mask(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
+    ) -> List[int]:
+        """
+        Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding
+        special tokens using the tokenizer ``prepare_for_model`` methods.
+
+        Args:
+            token_ids_0 (:obj:`List[int]`):
+                List of ids.
+            token_ids_1 (:obj:`List[int]`, `optional`):
+                Optional second list of IDs for sequence pairs.
+            already_has_special_tokens (:obj:`bool`, `optional`, defaults to :obj:`False`):
+                Set to True if the token list is already formatted with special tokens for the model
+
+        Returns:
+            :obj:`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
+        """
+
+        if already_has_special_tokens:
+            if token_ids_1 is not None:
+                raise ValueError(
+                    "You should not supply a second sequence if the provided sequence of "
+                    "ids is already formated with special tokens for the model."
+                )
+            return list(
+                map(
+                    lambda x: 1 if x in [self.sep_token_id, self.cls_token_id] else 0,
+                    token_ids_0,
+                )
+            )
+        # no bos used in fairseq
+        if token_ids_1 is not None:
+            return ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1]
+        return ([0] * len(token_ids_0)) + [1]
+
+    def create_token_type_ids_from_sequences(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Creates a mask from the two sequences passed to be used in a sequence-pair classification task.
+        An FAIRSEQ_TRANSFORMER sequence pair mask has the following format:
+
+        ::
+
+            0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
+            | first sequence    | second sequence |
+
+        if token_ids_1 is None, only returns the first portion of the mask (0s).
+
+        Args:
+            token_ids_0 (:obj:`List[int]`):
+                List of ids.
+            token_ids_1 (:obj:`List[int]`, `optional`):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            :obj:`List[int]`: List of `token type IDs <../glossary.html#token-type-ids>`_ according to the given
+            sequence(s).
+        """
+        sep = [self.sep_token_id]
+
+        # no bos used in fairseq
+        if token_ids_1 is None:
+            return len(token_ids_0 + sep) * [0]
+        return len(token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1]
+
+    @add_start_docstrings(PREPARE_SEQ2SEQ_BATCH_DOCSTRING)
+    def prepare_seq2seq_batch(
+        self,
+        src_texts: List[str],
+        tgt_texts: Optional[List[str]] = None,
+        max_length: Optional[int] = None,
+        max_target_length: Optional[int] = None,
+        return_tensors: str = "pt",
+        truncation=True,
+        padding="longest",
+        **unused,
+    ) -> BatchEncoding:
+        """Prepare model inputs for translation. For best performance, translate one sentence at a time."""
+
+        if type(src_texts) is not list:
+            raise ValueError("src_texts is expected to be a list")
+        if "" in src_texts:
+            raise ValueError(f"found empty string in src_texts: {src_texts}")
+
+        tokenizer_kwargs = dict(
+            add_special_tokens=True,
+            return_tensors=return_tensors,
+            max_length=max_length,
+            truncation=truncation,
+            padding=padding,
+        )
+        model_inputs: BatchEncoding = self(src_texts, **tokenizer_kwargs)
+
+        if tgt_texts is None:
+            return model_inputs
+        if max_target_length is not None:
+            tokenizer_kwargs["max_length"] = max_target_length
+
+        model_inputs["labels"] = self(tgt_texts, **tokenizer_kwargs)["input_ids"]
+        return model_inputs
+
+    def save_vocabulary(self, save_directory):
+        """
+        Save the vocabulary and special tokens file to a directory.
+
+        Args:
+            save_directory (:obj:`str`):
+                The directory in which to save the vocabulary.
+
+        Returns:
+            :obj:`Tuple(str)`: Paths to the files saved.
+        """
+        if not os.path.isdir(save_directory):
+            logger.error("Vocabulary path ({}) should be a directory".format(save_directory))
+            return
+
+        src_vocab_file = os.path.join(save_directory, VOCAB_FILES_NAMES["src_vocab_file"])
+        tgt_vocab_file = os.path.join(save_directory, VOCAB_FILES_NAMES["tgt_vocab_file"])
+        merges_file = os.path.join(save_directory, VOCAB_FILES_NAMES["merges_file"])
+
+        with open(src_vocab_file, "w", encoding="utf-8") as f:
+            f.write(json.dumps(self.encoder, ensure_ascii=False))
+
+        with open(tgt_vocab_file, "w", encoding="utf-8") as f:
+            tgt_vocab = {v: k for k, v in self.decoder.items()}
+            f.write(json.dumps(tgt_vocab, ensure_ascii=False))
+
+        index = 0
+        with open(merges_file, "w", encoding="utf-8") as writer:
+            for bpe_tokens, token_index in sorted(self.bpe_ranks.items(), key=lambda kv: kv[1]):
+                if index != token_index:
+                    logger.warning(
+                        "Saving vocabulary to {}: BPE merge indices are not consecutive."
+                        " Please check that the tokenizer is not corrupted!".format(merges_file)
+                    )
+                    index = token_index
+                writer.write(" ".join(bpe_tokens) + "\n")
+                index += 1
+
+        return src_vocab_file, tgt_vocab_file, merges_file

src/transformers/tokenization_pegasus.py

@@ -14,9 +14,8 @@
 # limitations under the License.
 from typing import Dict, List, Optional
 
-from transformers.tokenization_reformer import ReformerTokenizer
-
 from .file_utils import add_start_docstrings
+from .tokenization_reformer import ReformerTokenizer
 from .tokenization_utils_base import PREPARE_SEQ2SEQ_BATCH_DOCSTRING, BatchEncoding
 
 

src/transformers/tokenization_phobert.py

@@ -0,0 +1,369 @@
+# coding=utf-8
+# Copyright (c) 2020, VinAI Research and the HuggingFace Inc. team.
+# Copyright 2018 The Open AI Team Authors and The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Tokenization classes for PhoBERT """
+
+
+import os
+import re
+from shutil import copyfile
+from typing import List, Optional
+
+from .tokenization_utils import PreTrainedTokenizer
+from .utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+VOCAB_FILES_NAMES = {
+    "vocab_file": "vocab.txt",
+    "merges_file": "bpe.codes",
+}
+
+PRETRAINED_VOCAB_FILES_MAP = {
+    "vocab_file": {
+        "vinai/phobert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/vinai/phobert-base/vocab.txt",
+        "vinai/phobert-large": "https://s3.amazonaws.com/models.huggingface.co/bert/vinai/phobert-large/vocab.txt",
+    },
+    "merges_file": {
+        "vinai/phobert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/vinai/phobert-base/bpe.codes",
+        "vinai/phobert-large": "https://s3.amazonaws.com/models.huggingface.co/bert/vinai/phobert-large/bpe.codes",
+    },
+}
+
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
+    "vinai/phobert-base": 256,
+    "vinai/phobert-large": 256,
+}
+
+
+def get_pairs(word):
+    """Return set of symbol pairs in a word.
+
+    Word is represented as tuple of symbols (symbols being variable-length strings).
+    """
+    pairs = set()
+    prev_char = word[0]
+    for char in word[1:]:
+        pairs.add((prev_char, char))
+        prev_char = char
+
+    pairs = set(pairs)
+    return pairs
+
+
+class PhobertTokenizer(PreTrainedTokenizer):
+    """
+    Constructs a PhoBERT tokenizer. Peculiarities:
+
+    - Byte-Pair-Encoding
+
+    This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the methods. Users
+    should refer to the superclass for more information regarding methods.
+
+    Args:
+        vocab_file (:obj:`str`):
+            Path to the vocabulary file.
+        merges_file (:obj:`str`):
+            Path to the merges file.
+        bos_token (:obj:`string`, `optional`, defaults to "<s>"):
+            The beginning of sequence token that was used during pre-training. Can be used a sequence classifier token.
+
+            .. note::
+
+                When building a sequence using special tokens, this is not the token that is used for the beginning
+                of sequence. The token used is the :obj:`cls_token`.
+        eos_token (:obj:`string`, `optional`, defaults to "</s>"):
+            The end of sequence token.
+
+            .. note::
+
+                When building a sequence using special tokens, this is not the token that is used for the end
+                of sequence. The token used is the :obj:`sep_token`.
+        sep_token (:obj:`string`, `optional`, defaults to "</s>"):
+            The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences
+            for sequence classification or for a text and a question for question answering.
+            It is also used as the last token of a sequence built with special tokens.
+        cls_token (:obj:`string`, `optional`, defaults to "<s>"):
+            The classifier token which is used when doing sequence classification (classification of the whole
+            sequence instead of per-token classification). It is the first token of the sequence when built with
+            special tokens.
+        unk_token (:obj:`string`, `optional`, defaults to "<unk>"):
+            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
+            token instead.
+        pad_token (:obj:`string`, `optional`, defaults to "<pad>"):
+            The token used for padding, for example when batching sequences of different lengths.
+        mask_token (:obj:`string`, `optional`, defaults to "<mask>"):
+            The token used for masking values. This is the token used when training this model with masked language
+            modeling. This is the token which the model will try to predict.
+    """
+
+    vocab_files_names = VOCAB_FILES_NAMES
+    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
+    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+
+    def __init__(
+        self,
+        vocab_file,
+        merges_file,
+        bos_token="<s>",
+        eos_token="</s>",
+        sep_token="</s>",
+        cls_token="<s>",
+        unk_token="<unk>",
+        pad_token="<pad>",
+        mask_token="<mask>",
+        **kwargs
+    ):
+        super().__init__(
+            max_len=256,
+            bos_token=bos_token,
+            eos_token=eos_token,
+            unk_token=unk_token,
+            sep_token=sep_token,
+            cls_token=cls_token,
+            pad_token=pad_token,
+            mask_token=mask_token,
+            **kwargs,
+        )
+
+        self.vocab_file = vocab_file
+        self.merges_file = merges_file
+
+        self.encoder = {}
+        self.encoder[self.bos_token] = 0
+        self.encoder[self.pad_token] = 1
+        self.encoder[self.eos_token] = 2
+        self.encoder[self.unk_token] = 3
+
+        self.add_from_file(vocab_file)
+
+        self.decoder = {v: k for k, v in self.encoder.items()}
+
+        with open(merges_file, encoding="utf-8") as merges_handle:
+            merges = merges_handle.read().split("\n")[:-1]
+        merges = [tuple(merge.split()[:-1]) for merge in merges]
+        self.bpe_ranks = dict(zip(merges, range(len(merges))))
+        self.cache = {}
+
+    def build_inputs_with_special_tokens(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Build model inputs from a sequence or a pair of sequence for sequence classification tasks
+        by concatenating and adding special tokens.
+        A PhoBERT sequence has the following format:
+
+        - single sequence: ``<s> X </s>``
+        - pair of sequences: ``<s> A </s></s> B </s>``
+
+        Args:
+            token_ids_0 (:obj:`List[int]`):
+                List of IDs to which the special tokens will be added
+            token_ids_1 (:obj:`List[int]`, `optional`, defaults to :obj:`None`):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            :obj:`List[int]`: list of `input IDs <../glossary.html#input-ids>`__ with the appropriate special tokens.
+        """
+
+        if token_ids_1 is None:
+            return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
+        cls = [self.cls_token_id]
+        sep = [self.sep_token_id]
+        return cls + token_ids_0 + sep + sep + token_ids_1 + sep
+
+    def get_special_tokens_mask(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
+    ) -> List[int]:
+        """
+        Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding
+        special tokens using the tokenizer ``prepare_for_model`` methods.
+
+        Args:
+            token_ids_0 (:obj:`List[int]`):
+                List of ids.
+            token_ids_1 (:obj:`List[int]`, `optional`, defaults to :obj:`None`):
+                Optional second list of IDs for sequence pairs.
+            already_has_special_tokens (:obj:`bool`, `optional`, defaults to :obj:`False`):
+                Set to True if the token list is already formatted with special tokens for the model
+
+        Returns:
+            :obj:`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
+        """
+
+        if already_has_special_tokens:
+            if token_ids_1 is not None:
+                raise ValueError(
+                    "You should not supply a second sequence if the provided sequence of "
+                    "ids is already formated with special tokens for the model."
+                )
+            return list(map(lambda x: 1 if x in [self.sep_token_id, self.cls_token_id] else 0, token_ids_0))
+
+        if token_ids_1 is None:
+            return [1] + ([0] * len(token_ids_0)) + [1]
+        return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]
+
+    def create_token_type_ids_from_sequences(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Creates a mask from the two sequences passed to be used in a sequence-pair classification task.
+        PhoBERT does not make use of token type ids, therefore a list of zeros is returned.
+
+        Args:
+            token_ids_0 (:obj:`List[int]`):
+                List of ids.
+            token_ids_1 (:obj:`List[int]`, `optional`, defaults to :obj:`None`):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            :obj:`List[int]`: List of zeros.
+
+        """
+
+        sep = [self.sep_token_id]
+        cls = [self.cls_token_id]
+
+        if token_ids_1 is None:
+            return len(cls + token_ids_0 + sep) * [0]
+        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]
+
+    @property
+    def vocab_size(self):
+        return len(self.encoder)
+
+    def get_vocab(self):
+        return dict(self.encoder, **self.added_tokens_encoder)
+
+    def bpe(self, token):
+        if token in self.cache:
+            return self.cache[token]
+        word = tuple(token)
+        word = tuple(list(word[:-1]) + [word[-1] + "</w>"])
+        pairs = get_pairs(word)
+
+        if not pairs:
+            return token
+
+        while True:
+            bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf")))
+            if bigram not in self.bpe_ranks:
+                break
+            first, second = bigram
+            new_word = []
+            i = 0
+            while i < len(word):
+                try:
+                    j = word.index(first, i)
+                except ValueError:
+                    new_word.extend(word[i:])
+                    break
+                else:
+                    new_word.extend(word[i:j])
+                    i = j
+
+                if word[i] == first and i < len(word) - 1 and word[i + 1] == second:
+                    new_word.append(first + second)
+                    i += 2
+                else:
+                    new_word.append(word[i])
+                    i += 1
+            new_word = tuple(new_word)
+            word = new_word
+            if len(word) == 1:
+                break
+            else:
+                pairs = get_pairs(word)
+        word = "@@ ".join(word)
+        word = word[:-4]
+        self.cache[token] = word
+        return word
+
+    def _tokenize(self, text):
+        """Tokenize a string."""
+        split_tokens = []
+
+        words = re.findall(r"\S+\n?", text)
+
+        for token in words:
+            split_tokens.extend([t for t in self.bpe(token).split(" ")])
+        return split_tokens
+
+    def _convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        return self.encoder.get(token, self.encoder.get(self.unk_token))
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        return self.decoder.get(index, self.unk_token)
+
+    def convert_tokens_to_string(self, tokens):
+        """ Converts a sequence of tokens (string) in a single string. """
+        out_string = " ".join(tokens).replace("@@ ", "").strip()
+        return out_string
+
+    def save_vocabulary(self, save_directory):
+        """
+        Save the vocabulary and special tokens file to a directory.
+        Args:
+            save_directory (:obj:`str`):
+                The directory in which to save the vocabulary.
+        Returns:
+            :obj:`Tuple(str)`: Paths to the files saved.
+        """
+        if not os.path.isdir(save_directory):
+            logger.error("Vocabulary path ({}) should be a directory".format(save_directory))
+            return
+        out_vocab_file = os.path.join(save_directory, VOCAB_FILES_NAMES["vocab_file"])
+        out_merge_file = os.path.join(save_directory, VOCAB_FILES_NAMES["merges_file"])
+
+        if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
+            copyfile(self.vocab_file, out_vocab_file)
+
+        if os.path.abspath(self.merges_file) != os.path.abspath(out_merge_file):
+            copyfile(self.merges_file, out_merge_file)
+
+        return out_vocab_file, out_merge_file
+
+    # def decode(self, token_ids, skip_special_tokens=False, clean_up_tokenization_spaces=True):
+    #     filtered_tokens = ' '.join(self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens))
+    #     tokens_generated_so_far = re.sub('(@@ )', '', string=filtered_tokens)
+    #     tokens_generated_so_far = re.sub('(@@ ?$)', '', string=tokens_generated_so_far)
+    #     return ''.join(tokens_generated_so_far)
+
+    def add_from_file(self, f):
+        """
+        Loads a pre-existing dictionary from a text file and adds its symbols
+        to this instance.
+        """
+        if isinstance(f, str):
+            try:
+                with open(f, "r", encoding="utf-8") as fd:
+                    self.add_from_file(fd)
+            except FileNotFoundError as fnfe:
+                raise fnfe
+            except UnicodeError:
+                raise Exception("Incorrect encoding detected in {}, please " "rebuild the dataset".format(f))
+            return
+
+        lines = f.readlines()
+        for lineTmp in lines:
+            line = lineTmp.strip()
+            idx = line.rfind(" ")
+            if idx == -1:
+                raise ValueError("Incorrect dictionary format, expected '<token> <cnt>'")
+            word = line[:idx]
+            self.encoder[word] = len(self.encoder)