mbart - marian - mobilebert - openai

longformer - lxmert
herbert and layoutlm
2020-11-25 23:52:09 +01:00 · 2020-11-25 19:30:33 +01:00 · 2020-11-25 19:14:55 +01:00 · 2020-11-25 19:03:07 +01:00 · 2020-11-25 18:49:18 +01:00 · 2020-11-25 18:36:19 +01:00
275 changed files with 2329 additions and 2597 deletions
@@ -221,7 +221,7 @@ jobs:
    run_tests_custom_tokenizers:
        working_directory: ~/transformers
        docker:
-            - image: circleci/python:3.6
+            - image: circleci/python:3.7
        environment:
            RUN_CUSTOM_TOKENIZERS: yes
        steps:
@@ -52,5 +52,4 @@ deploy_doc "4b3ee9c" v3.1.0
 deploy_doc "3ebb1b3" v3.2.0
 deploy_doc "0613f05" v3.3.1
 deploy_doc "eb0e0ce" v3.4.0
-deploy_doc "818878d" v3.5.1
-deploy_doc "28d3ccd" # v4.0.1 Latest stable release
+deploy_doc "818878d" # v3.5.1 Latest stable release
@@ -1 +0,0 @@
-$PYTHON setup.py install     # Python command to install the script.
@@ -1,48 +0,0 @@
-{% set name = "transformers" %}
-
-package:
-  name: "{{ name|lower }}"
-  version: "{{ TRANSFORMERS_VERSION }}"
-
-source:
-  path: ../../
-
-build:
-  noarch: python
-
-requirements:
-  host:
-    - python
-    - pip
-    - numpy
-    - dataclasses
-    - packaging
-    - filelock
-    - requests
-    - tqdm >=4.27
-    - sacremoses
-    - regex !=2019.12.17
-    - protobuf
-    - tokenizers ==0.9.4
-  run:
-    - python
-    - numpy
-    - dataclasses
-    - packaging
-    - filelock
-    - requests
-    - tqdm >=4.27
-    - sacremoses
-    - regex !=2019.12.17
-    - protobuf
-    - tokenizers ==0.9.4
-
-test:
-  imports:
-    - transformers
-
-about:
-  home: https://huggingface.co
-  license: Apache License 2.0
-  license_file: LICENSE
-  summary: "🤗Transformers: State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0."
@@ -1,43 +0,0 @@
-name: Release - Conda
-
-on:
-  push:
-    tags:
-      - v*
-
-env:
-  ANACONDA_API_TOKEN: ${{ secrets.ANACONDA_API_TOKEN }}
-
-jobs:
-  build_and_package:
-    runs-on: ubuntu-latest
-    defaults:
-      run:
-        shell: bash -l {0}
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v1
-
-      - name: Install miniconda
-        uses: conda-incubator/setup-miniconda@v2
-        with:
-          auto-update-conda: true
-          auto-activate-base: false
-          activate-environment: "build-transformers"
-          channels: huggingface
-
-      - name: Setup conda env
-        run: |
-          conda install -c defaults anaconda-client conda-build
-
-      - name: Extract version
-        run: echo "TRANSFORMERS_VERSION=`python setup.py --version`" >> $GITHUB_ENV
-
-      - name: Build conda packages
-        run: |
-          conda info
-          conda build .github/conda
-
-      - name: Upload to Anaconda
-        run: anaconda upload `conda build .github/conda --output` --force
@@ -9,6 +9,7 @@ on:
  push:
    branches:
      - ci_*
+      - framework-agnostic-tokenizers
  repository_dispatch:
  schedule:
    - cron: "0 0 * * *"
@@ -137,16 +137,14 @@ The model itself is a regular [Pytorch `nn.Module`](https://pytorch.org/docs/sta

 ## Installation

-### With pip
-
 This repository is tested on Python 3.6+, PyTorch 1.0.0+ (PyTorch 1.3.1+ for [examples](https://github.com/huggingface/transformers/tree/master/examples)) and TensorFlow 2.0.

 You should install 🤗 Transformers in a [virtual environment](https://docs.python.org/3/library/venv.html). If you're unfamiliar with Python virtual environments, check out the [user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).

 First, create a virtual environment with the version of Python you're going to use and activate it.

-Then, you will need to install at least one of TensorFlow 2.0, PyTorch or Flax.
-Please refer to [TensorFlow installation page](https://www.tensorflow.org/install/pip#tensorflow-2.0-rc-is-available), [PyTorch installation page](https://pytorch.org/get-started/locally/#start-locally) regarding the specific install command for your platform and/or [Flax installation page](https://github.com/google/flax#quick-install).
+Then, you will need to install one of, or both, TensorFlow 2.0 and PyTorch.
+Please refer to [TensorFlow installation page](https://www.tensorflow.org/install/pip#tensorflow-2.0-rc-is-available) and/or [PyTorch installation page](https://pytorch.org/get-started/locally/#start-locally) regarding the specific install command for your platform.

 When TensorFlow 2.0 and/or PyTorch has been installed, 🤗 Transformers can be installed using pip as follows:

@@ -156,18 +154,6 @@ pip install transformers

 If you'd like to play with the examples, you must [install the library from source](https://huggingface.co/transformers/installation.html#installing-from-source).

-### With conda
-
-Since Transformers version v4.0.0, we now have a conda channel: `huggingface`.
-
-🤗 Transformers can be installed using conda as follows:
-
-```shell script
-conda install -c huggingface transformers
-```
-
-Follow the installation pages of TensorFlow, PyTorch or Flax to see how to install them with conda. 
-
 ## Models architectures

 🤗 Transformers currently provides the following architectures (see [here](https://huggingface.co/transformers/model_summary.html) for a high-level summary of each them):
@@ -211,8 +197,6 @@ ultilingual BERT into [DistilmBERT](https://github.com/huggingface/transformers/
 1. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users).
 1. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.

-To cehck if each model has an implementation in PyTorch/TensorFlow/Flax or has an associated tokenizer backed by the 🤗 Tokenizers library, refer to [this table](https://huggingface.co/transformers/index.html#bigtable)
-
 These implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations. You can find more details on the performances in the Examples section of the [documentation](https://huggingface.co/transformers/examples.html).


@@ -230,13 +214,17 @@ These implementations have been tested on several datasets (see the example scri

 ## Citation

-We now have a [paper](https://arxiv.org/abs/1910.03771) you can cite for the 🤗 Transformers library:
+We now have a [paper](https://www.aclweb.org/anthology/2020.emnlp-demos.6/) you can cite for the 🤗 Transformers library:
 ```bibtex
-@article{Wolf2019HuggingFacesTS,
-  title={HuggingFace's Transformers: State-of-the-art Natural Language Processing},
-  author={Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Joe Davison and Sam Shleifer and Patrick von Platen and Clara Ma and Yacine Jernite and Julien Plu and Canwen Xu and Teven Le Scao and Sylvain Gugger and Mariama Drame and Quentin Lhoest and Alexander M. Rush},
-  journal={ArXiv},
-  year={2019},
-  volume={abs/1910.03771}
+@inproceedings{wolf-etal-2020-transformers,
+    title = "Transformers: State-of-the-Art Natural Language Processing",
+    author = "Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Joe Davison and Sam Shleifer and Patrick von Platen and Clara Ma and Yacine Jernite and Julien Plu and Canwen Xu and Teven Le Scao and Sylvain Gugger and Mariama Drame and Quentin Lhoest and Alexander M. Rush",
+    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
+    month = oct,
+    year = "2020",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://www.aclweb.org/anthology/2020.emnlp-demos.6",
+    pages = "38--45"
 }
 ```
@@ -2,15 +2,6 @@

 /* Colab dropdown */

-table.center-aligned-table td {
-    text-align: center;
-}
-
-table.center-aligned-table th {
-    text-align: center;
-    vertical-align: middle;
-}
-
 .colab-dropdown {
    position: relative;
    display: inline-block;
@@ -1,15 +1,14 @@
 // These two things need to be updated at each release for the version selector.
 // Last stable version
-const stableVersion = "v4.0.1"
+const stableVersion = "v3.5.0"
 // Dictionary doc folder to label
 const versionMapping = {
    "master": "master",
-    "": "v4.0.0/v4.0.1",
-    "v3.5.1": "v3.5.0/v3.5.1",
+    "": "v3.5.0/v3.5.1",
    "v3.4.0": "v3.4.0",
    "v3.3.1": "v3.3.0/v3.3.1",
    "v3.2.0": "v3.2.0",
-    "v3.1.0": "v3.1.0",
+    "v3.1.0": "v3.1.0 (stable)",
    "v3.0.2": "v3.0.0/v3.0.1/v3.0.2",
    "v2.11.0": "v2.11.0",
    "v2.10.0": "v2.10.0",
@@ -26,7 +26,7 @@ author = u'huggingface'
 # The short X.Y version
 version = u''
 # The full version, including alpha/beta/rc tags
-release = u'4.0.1'
+release = u'3.5.0'


 # -- General configuration ---------------------------------------------------
@@ -35,8 +35,6 @@ Choose the right framework for every part of a model's lifetime:
 - Move a single model between TF2.0/PyTorch frameworks at will
 - Seamlessly pick the right framework for training, evaluation, production

-Experimental support for Flax with a few models right now, expected to grow in the coming months.
-
 Contents
 -----------------------------------------------------------------------------------------------------------------------

@@ -54,8 +52,8 @@ The documentation is organized in five parts:
    - **MODELS** for the classes and functions related to each model implemented in the library.
    - **INTERNAL HELPERS** for the classes and functions we use internally.

-The library currently contains PyTorch, Tensorflow and Flax implementations, pretrained model weights, usage scripts
-and conversion utilities for the following models:
+The library currently contains PyTorch and Tensorflow implementations, pre-trained model weights, usage scripts and
+conversion utilities for the following models:

 ..
    This list is updated automatically from the README with `make fix-copies`. Do not update manually!
@@ -168,97 +166,6 @@ and conversion utilities for the following models:
 34. `Other community models <https://huggingface.co/models>`__, contributed by the `community
    <https://huggingface.co/users>`__.

-
-.. _bigtable:
-
-The table below represents the current support in the library for each of those models, whether they have a Python
-tokenizer (called "slow"). A "fast" tokenizer backed by the 🤗 Tokenizers library, whether they have support in PyTorch,
-TensorFlow and/or Flax.
-
-..
-    This table is updated automatically from the auto modules with `make fix-copies`. Do not update manually!
-
-.. rst-class:: center-aligned-table
-
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|            Model            | Tokenizer slow | Tokenizer fast | PyTorch support | TensorFlow support | Flax Support |
-+=============================+================+================+=================+====================+==============+
-|           ALBERT            |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|            BART             |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|            BERT             |       ✅       |       ✅       |       ✅        |         ✅         |      ✅      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|       Bert Generation       |       ✅       |       ❌       |       ✅        |         ❌         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|         Blenderbot          |       ✅       |       ❌       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|            CTRL             |       ✅       |       ❌       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|          CamemBERT          |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|             DPR             |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|           DeBERTa           |       ✅       |       ❌       |       ✅        |         ❌         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|         DistilBERT          |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|           ELECTRA           |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|       Encoder decoder       |       ❌       |       ❌       |       ✅        |         ❌         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-| FairSeq Machine-Translation |       ✅       |       ❌       |       ✅        |         ❌         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|          FlauBERT           |       ✅       |       ❌       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|     Funnel Transformer      |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|           LXMERT            |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|          LayoutLM           |       ✅       |       ✅       |       ✅        |         ❌         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|         Longformer          |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|           Marian            |       ✅       |       ❌       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|         MobileBERT          |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|         OpenAI GPT          |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|        OpenAI GPT-2         |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|           Pegasus           |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|         ProphetNet          |       ✅       |       ❌       |       ✅        |         ❌         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|             RAG             |       ✅       |       ❌       |       ✅        |         ❌         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|          Reformer           |       ✅       |       ✅       |       ✅        |         ❌         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|          RetriBERT          |       ✅       |       ✅       |       ✅        |         ❌         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|           RoBERTa           |       ✅       |       ✅       |       ✅        |         ✅         |      ✅      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|         SqueezeBERT         |       ✅       |       ✅       |       ✅        |         ❌         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|             T5              |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|       Transformer-XL        |       ✅       |       ❌       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|             XLM             |       ✅       |       ❌       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|         XLM-RoBERTa         |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|        XLMProphetNet        |       ✅       |       ❌       |       ✅        |         ❌         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|            XLNet            |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|            mBART            |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-|             mT5             |       ✅       |       ✅       |       ✅        |         ✅         |      ❌      |
-+-----------------------------+----------------+----------------+-----------------+--------------------+--------------+
-
-
 .. toctree::
    :maxdepth: 2
    :caption: Get started
@@ -12,10 +12,9 @@ must install it from source.
 ## Installation with pip

 First you need to install one of, or both, TensorFlow 2.0 and PyTorch.
-Please refer to [TensorFlow installation page](https://www.tensorflow.org/install/pip#tensorflow-2.0-rc-is-available), 
-[PyTorch installation page](https://pytorch.org/get-started/locally/#start-locally) and/or 
-[Flax installation page](https://github.com/google/flax#quick-install)
-regarding the specific install command for your platform.
+Please refer to [TensorFlow installation page](https://www.tensorflow.org/install/pip#tensorflow-2.0-rc-is-available) 
+and/or [PyTorch installation page](https://pytorch.org/get-started/locally/#start-locally) regarding the specific 
+install command for your platform.

 When TensorFlow 2.0 and/or PyTorch has been installed, 🤗 Transformers can be installed using pip as follows:

@@ -35,12 +34,6 @@ or 🤗 Transformers and TensorFlow 2.0 in one line with:
 pip install transformers[tf-cpu]
 ```

-or 🤗 Transformers and Flax in one line with:
-
-```bash
-pip install transformers[flax]
-```
-
 To check 🤗 Transformers is properly installed, run the following command:

 ```bash
@@ -73,19 +66,6 @@ python -c "from transformers import pipeline; print(pipeline('sentiment-analysis

 to check 🤗 Transformers is properly installed.

-
-## With conda
-
-Since Transformers version v4.0.0, we now have a conda channel: `huggingface`.
-
-🤗 Transformers can be installed using conda as follows:
-
-```
-conda install -c huggingface transformers
-```
-
-Follow the installation pages of TensorFlow, PyTorch or Flax to see how to install them with conda. 
-
 ## Caching models

 This library provides pretrained models that will be downloaded and cached locally. Unless you specify a location with
@@ -117,6 +97,6 @@ You should check out our [swift-coreml-transformers](https://github.com/huggingf
 It contains a set of tools to convert PyTorch or TensorFlow 2.0 trained Transformer models (currently contains `GPT-2`, 
 `DistilGPT-2`, `BERT`, and `DistilBERT`) to CoreML models that run on iOS devices.

-At some point in the future, you'll be able to seamlessly move from pretraining or fine-tuning models in PyTorch or
+At some point in the future, you'll be able to seamlessly move from pre-training or fine-tuning models in PyTorch or
 TensorFlow 2.0 to productizing them in CoreML, or prototype a model or an app in CoreML then research its
 hyperparameters or architecture from PyTorch or TensorFlow 2.0. Super exciting!
@@ -1,170 +1,5 @@
 # Migrating from previous packages

-## Migrating from transformers `v3.x` to `v4.x`
-
-A couple of changes were introduced when the switch from version 3 to version 4 was done. Below is a summary of the
-expected changes:
-
-#### 1. AutoTokenizers and pipelines now use fast (rust) tokenizers by default.
-
-The python and rust tokenizers have roughly the same API, but the rust tokenizers have a more complete feature set. 
-
-This introduces two breaking changes:
- The handling of overflowing tokens between the python and rust tokenizers is different.
- The rust tokenizers do not accept integers in the encoding methods.
-
-##### How to obtain the same behavior as v3.x in v4.x
-
- The pipelines now contain additional features out of the box. See the [token-classification pipeline with the `grouped_entities` flag](https://huggingface.co/transformers/main_classes/pipelines.html?highlight=textclassification#tokenclassificationpipeline).
- The auto-tokenizers now return rust tokenizers. In order to obtain the python tokenizers instead, the user may use the `use_fast` flag by setting it to `False`:
-
-In version `v3.x`:
-```py
-from transformers import AutoTokenizer
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-```
-to obtain the same in version `v4.x`:
-```py
-from transformers import AutoTokenizer
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased", use_fast=False)
-```
-
-#### 2. SentencePiece is removed from the required dependencies
-
-The requirement on the SentencePiece dependency has been lifted from the `setup.py`. This is done so that we may have a channel on anaconda cloud without relying on `conda-forge`. This means that the tokenizers that depend on the SentencePiece library will not be available with a standard `transformers` installation.
-
-This includes the **slow** versions of:
- `XLNetTokenizer`
- `AlbertTokenizer`
- `CamembertTokenizer`
- `MBartTokenizer`
- `PegasusTokenizer`
- `T5Tokenizer`
- `ReformerTokenizer`
- `XLMRobertaTokenizer`
-
-##### How to obtain the same behavior as v3.x in v4.x
-
-In order to obtain the same behavior as version `v3.x`, you should install `sentencepiece` additionally:
-
-In version `v3.x`:
-```bash
-pip install transformers
-```
-to obtain the same in version `v4.x`:
-```bash
-pip install transformers[sentencepiece]
-```
-or
-```bash
-pip install transformers sentencepiece
-```
-#### 3. The architecture of the repo has been updated so that each model resides in its folder
-
-The past and foreseeable addition of new models means that the number of files in the directory `src/transformers` keeps growing and becomes harder to navigate and understand. We made the choice to put each model and the files accompanying it in their own sub-directories.
-
-This is a breaking change as importing intermediary layers using a model's module directly needs to be done via a different path.
-
-##### How to obtain the same behavior as v3.x in v4.x
-
-In order to obtain the same behavior as version `v3.x`, you should update the path used to access the layers. 
-
-In version `v3.x`:
-```bash
-from transformers.modeling_bert import BertLayer
-```
-to obtain the same in version `v4.x`:
-```bash
-from transformers.models.bert.modeling_bert import BertLayer
-```
-
-#### 4. Switching the `return_dict` argument to `True` by default
-
-The [`return_dict` argument](https://huggingface.co/transformers/main_classes/output.html) enables the return of dict-like python objects containing the model outputs, instead of the standard tuples. This object is self-documented as keys can be used to retrieve values, while also behaving as a tuple as users may retrieve objects by index or by slice.
-
-This is a breaking change as the limitation of that tuple is that it cannot be unpacked: `value0, value1 = outputs` will not work.
-
-##### How to obtain the same behavior as v3.x in v4.x
-
-In order to obtain the same behavior as version `v3.x`, you should specify the `return_dict` argument to `False`, either in the model configuration or during the forward pass.
-
-In version `v3.x`:
-```bash
-model = BertModel.from_pretrained("bert-base-cased")
-outputs = model(**inputs)
-```
-to obtain the same in version `v4.x`:
-```bash
-model = BertModel.from_pretrained("bert-base-cased")
-outputs = model(**inputs, return_dict=False)
-```
-or
-```bash
-model = BertModel.from_pretrained("bert-base-cased", return_dict=False)
-outputs = model(**inputs)
-```
-
-#### 5. Removed some deprecated attributes
-
-Attributes that were deprecated have been removed if they had been deprecated for at least a month. The full list of deprecated attributes can be found in [#8604](https://github.com/huggingface/transformers/pull/8604).
-
-Here is a list of these attributes/methods/arguments and what their replacements should be:
-
-In several models, the labels become consistent with the other models:
- `masked_lm_labels` becomes `labels` in `AlbertForMaskedLM` and `AlbertForPreTraining`.
- `masked_lm_labels` becomes `labels` in `BertForMaskedLM` and `BertForPreTraining`.
- `masked_lm_labels` becomes `labels` in `DistilBertForMaskedLM`.
- `masked_lm_labels` becomes `labels` in `ElectraForMaskedLM`.
- `masked_lm_labels` becomes `labels` in `LongformerForMaskedLM`.
- `masked_lm_labels` becomes `labels` in `MobileBertForMaskedLM`.
- `masked_lm_labels` becomes `labels` in `RobertaForMaskedLM`.
- `lm_labels` becomes `labels` in `BartForConditionalGeneration`.
- `lm_labels` becomes `labels` in `GPT2DoubleHeadsModel`.
- `lm_labels` becomes `labels` in `OpenAIGPTDoubleHeadsModel`.
- `lm_labels` becomes `labels` in `T5ForConditionalGeneration`.
-
-In several models, the caching mechanism becomes consistent with the other models:
- `decoder_cached_states` becomes `past_key_values` in all BART-like, FSMT and T5 models.
- `decoder_past_key_values` becomes `past_key_values` in all BART-like, FSMT and T5 models.
- `past` becomes `past_key_values` in all CTRL models.
- `past` becomes `past_key_values` in all GPT-2 models.
-
-Regarding the tokenizer classes:
- The tokenizer attribute `max_len` becomes `model_max_length`.
- The tokenizer attribute `return_lengths` becomes `return_length`.
- The tokenizer encoding argument `is_pretokenized` becomes `is_split_into_words`.
-
-Regarding the `Trainer` class:
- The `Trainer` argument `tb_writer` is removed in favor of the callback `TensorBoardCallback(tb_writer=...)`.
- The `Trainer` argument `prediction_loss_only` is removed in favor of the class argument `args.prediction_loss_only`.
- The `Trainer` attribute `data_collator` should be a callable.
- The `Trainer` method `_log` is deprecated in favor of `log`.
- The `Trainer` method `_training_step` is deprecated in favor of `training_step`.
- The `Trainer` method `_prediction_loop` is deprecated in favor of `prediction_loop`.
- The `Trainer` method `is_local_master` is deprecated in favor of `is_local_process_zero`.
- The `Trainer` method `is_world_master` is deprecated in favor of `is_world_process_zero`.
-
-Regarding the `TFTrainer` class:
- The `TFTrainer` argument `prediction_loss_only` is removed in favor of the class argument `args.prediction_loss_only`.
- The `Trainer` method `_log` is deprecated in favor of `log`.
- The `TFTrainer` method `_prediction_loop` is deprecated in favor of `prediction_loop`.
- The `TFTrainer` method `_setup_wandb` is deprecated in favor of `setup_wandb`.
- The `TFTrainer` method `_run_model` is deprecated in favor of `run_model`.
-
-Regarding the `TrainerArgument` class:
- The `TrainerArgument` argument `evaluate_during_training` is deprecated in favor of `evaluation_strategy`.
-
-Regarding the Transfo-XL model:
- The Transfo-XL configuration attribute `tie_weight` becomes `tie_words_embeddings`.
- The Transfo-XL modeling method `reset_length` becomes `reset_memory_length`.
-
-Regarding pipelines:
- The `FillMaskPipeline` argument `topk` becomes `top_k`.
-
-
-
 ## Migrating from pytorch-transformers to 🤗 Transformers

 Here is a quick summary of what you should take care of when migrating from `pytorch-transformers` to 🤗 Transformers.
@@ -10,7 +10,7 @@ Tasks <https://arxiv.org/abs/1907.12461>`__ by Sascha Rothe, Shashi Narayan, Ali

 The abstract from the paper is the following:

-*Unsupervised pretraining of large neural models has recently revolutionized Natural Language Processing. By
+*Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing. By
 warm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple
 benchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language
 Understanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We
@@ -20,8 +20,8 @@ disentangled attention mechanism, where each word is represented using two vecto
 position, respectively, and the attention weights among words are computed using disentangled matrices on their
 contents and relative positions. Second, an enhanced mask decoder is used to replace the output softmax layer to
 predict the masked tokens for model pretraining. We show that these two techniques significantly improve the efficiency
-of model pretraining and performance of downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of
-the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9%
+of model pre-training and performance of downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half
+of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9%
 (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). The DeBERTa code and
 pre-trained models will be made publicly available at https://github.com/microsoft/DeBERTa.*

@@ -18,9 +18,9 @@ operating these large models in on-the-edge and/or under constrained computation
 remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation
 model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger
 counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage
-knowledge distillation during the pretraining phase and show that it is possible to reduce the size of a BERT model by
+knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by
 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive
-biases learned by larger models during pretraining, we introduce a triple loss combining language modeling,
+biases learned by larger models during pre-training, we introduce a triple loss combining language modeling,
 distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we
 demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device
 study.*
@@ -12,14 +12,14 @@ identify which tokens were replaced by the generator in the sequence.

 The abstract from the paper is the following:

-*Masked language modeling (MLM) pretraining methods such as BERT corrupt the input by replacing some tokens with [MASK]
-and then train a model to reconstruct the original tokens. While they produce good results when transferred to
+*Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with
+[MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to
 downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a
-more sample-efficient pretraining task called replaced token detection. Instead of masking the input, our approach
+more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach
 corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead
 of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that
 predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments
-demonstrate this new pretraining task is more efficient than MLM because the task is defined over all input tokens
+demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens
 rather than just the small subset that was masked out. As a result, the contextual representations learned by our
 approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are
 particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained
@@ -19,7 +19,7 @@ representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018;
 heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for
 Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text
 classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the
-time they outperform other pretraining approaches. Different versions of FlauBERT as well as a unified evaluation
+time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation
 protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research
 community for further reproducible experiments in French NLP.*

@@ -14,7 +14,7 @@ The abstract from the paper is the following:
 *Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering,
 semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant,
 labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to
-perform adequately. We demonstrate that large gains on these tasks can be realized by generative pretraining of a
+perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a
 language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In
 contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve
 effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our
@@ -6,19 +6,19 @@ Overview

 The LayoutLM model was proposed in the paper `LayoutLM: Pre-training of Text and Layout for Document Image
 Understanding <https://arxiv.org/abs/1912.13318>`__ by Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and
-Ming Zhou. It's a simple but effective pretraining method of text and layout for document image understanding and
+Ming Zhou. It's a simple but effective pre-training method of text and layout for document image understanding and
 information extraction tasks, such as form understanding and receipt understanding.

 The abstract from the paper is the following:

 *Pre-training techniques have been verified successfully in a variety of NLP tasks in recent years. Despite the
-widespread use of pretraining models for NLP applications, they almost exclusively focus on text-level manipulation,
+widespread use of pre-training models for NLP applications, they almost exclusively focus on text-level manipulation,
 while neglecting layout and style information that is vital for document image understanding. In this paper, we propose
 the \textbf{LayoutLM} to jointly model interactions between text and layout information across scanned document images,
 which is beneficial for a great number of real-world document image understanding tasks such as information extraction
 from scanned documents. Furthermore, we also leverage image features to incorporate words' visual information into
 LayoutLM. To the best of our knowledge, this is the first time that text and layout are jointly learned in a single
-framework for document-level pretraining. It achieves new state-of-the-art results in several downstream tasks,
+framework for document-level pre-training. It achieves new state-of-the-art results in several downstream tasks,
 including form understanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image
 classification (from 93.07 to 94.42).*

@@ -19,7 +19,7 @@ Encoder Representations from Transformers) framework to learn these vision-and-l
 build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language
 encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language
 semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative
-pretraining tasks: masked language modeling, masked object prediction (feature regression and label classification),
+pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification),
 cross-modality matching, and image question answering. These tasks help in learning both intra-modality and
 cross-modality relationships. After fine-tuning from our pretrained parameters, our model achieves the state-of-the-art
 results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our
@@ -13,7 +13,7 @@ The MBart model was presented in `Multilingual Denoising Pre-training for Neural
 Ghazvininejad, Mike Lewis, Luke Zettlemoyer.

 According to the abstract, MBART is a sequence-to-sequence denoising auto-encoder pretrained on large-scale monolingual
-corpora in many languages using the BART objective. mBART is one of the first methods for pretraining a complete
+corpora in many languages using the BART objective. mBART is one of the first methods for pre-training a complete
 sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only
 on the encoder, decoder, or reconstructing parts of the text.

@@ -17,7 +17,7 @@ the next token.

 The abstract from the paper is the following:

-*In this paper, we present a new sequence-to-sequence pretraining model called ProphetNet, which introduces a novel
+*In this paper, we present a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel
 self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of
 the optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by
 n-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time
@@ -25,7 +25,7 @@ step. The future n-gram prediction explicitly encourages the model to plan for t
 overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale
 dataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for
 abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new
-state-of-the-art results on all these datasets compared to the models using the same scale pretraining corpus.*
+state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.*

 The Authors' code can be found `here <https://github.com/microsoft/ProphetNet>`__.

@@ -17,7 +17,7 @@ The abstract from the paper is the following:
 task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning
 has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of
 transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a
-text-to-text format. Our systematic study compares pretraining objectives, architectures, unlabeled datasets, transfer
+text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer
 approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration
 with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering
 summarization, question answering, text classification, and more. To facilitate future work on transfer learning for
@@ -19,7 +19,7 @@ just the next token. Its architecture is identical to ProhpetNet, but the model

 The abstract from the paper is the following:

-*In this paper, we present a new sequence-to-sequence pretraining model called ProphetNet, which introduces a novel
+*In this paper, we present a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel
 self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of
 the optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by
 n-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time
@@ -27,7 +27,7 @@ step. The future n-gram prediction explicitly encourages the model to plan for t
 overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale
 dataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for
 abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new
-state-of-the-art results on all these datasets compared to the models using the same scale pretraining corpus.*
+state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.*

 The Authors' code can be found `here <https://github.com/microsoft/ProphetNet>`__.

@@ -527,7 +527,7 @@ Pegasus
 <https://arxiv.org/pdf/1912.08777.pdf>`_, Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019.

 Sequence-to-sequence model with the same encoder-decoder model architecture as BART. Pegasus is pre-trained jointly on
-two self-supervised objective functions: Masked Language Modeling (MLM) and a novel summarization specific pretraining
+two self-supervised objective functions: Masked Language Modeling (MLM) and a novel summarization specific pre-training
 objective, called Gap Sentence Generation (GSG).

  * MLM: encoder input tokens are randomly replaced by a mask tokens and have to be predicted by the encoder (like in
@@ -609,7 +609,7 @@ MT5
 `mT5: A massively multilingual pre-trained text-to-text transformer <https://arxiv.org/abs/2010.11934>`_, Linting Xue
 et al.

-The model architecture is same as T5. mT5's pretraining objective includes T5's self-supervised training, but not T5's
+The model architecture is same as T5. mT5's pre-training objective includes T5's self-supervised training, but not T5's
 supervised training. mT5 is trained on 101 languages.

 The library provides a version of this model for conditional generation.
@@ -630,8 +630,8 @@ MBart
 `Multilingual Denoising Pre-training for Neural Machine Translation <https://arxiv.org/abs/2001.08210>`_ by Yinhan Liu,
 Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.

-The model architecture and pretraining objective is same as BART, but MBart is trained on 25 languages and is intended
-for supervised and unsupervised machine translation. MBart is one of the first methods for pretraining a complete
+The model architecture and pre-training objective is same as BART, but MBart is trained on 25 languages and is intended
+for supervised and unsupervised machine translation. MBart is one of the first methods for pre-training a complete
 sequence-to-sequence model by denoising full texts in multiple languages,

 The library provides a version of this model for conditional generation.
@@ -658,7 +658,7 @@ ProphetNet
 `ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training, <https://arxiv.org/abs/2001.04063>`__ by
 Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, Ming Zhou.

-ProphetNet introduces a novel *sequence-to-sequence* pretraining objective, called *future n-gram prediction*. In
+ProphetNet introduces a novel *sequence-to-sequence* pre-training objective, called *future n-gram prediction*. In
 future n-gram prediction, the model predicts the next n tokens simultaneously based on previous context tokens at each
 time step instead instead of just the single next token. The future n-gram prediction explicitly encourages the model
 to plan for the future tokens and prevent overfitting on strong local correlations. The model architecture is based on
@@ -683,8 +683,8 @@ XLM-ProphetNet
 `ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training, <https://arxiv.org/abs/2001.04063>`__ by
 Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, Ming Zhou.

-XLM-ProphetNet's model architecture and pretraining objective is same as ProphetNet, but XLM-ProphetNet was pre-trained
-on the cross-lingual dataset `XGLUE <https://arxiv.org/abs/2004.01401>`__.
+XLM-ProphetNet's model architecture and pre-training objective is same as ProphetNet, but XLM-ProphetNet was
+pre-trained on the cross-lingual dataset `XGLUE <https://arxiv.org/abs/2004.01401>`__.

 The library provides a pre-trained version of this model for multi-lingual conditional generation and fine-tuned
 versions for headline generation and question generation, respectively.
@@ -305,7 +305,7 @@ Language modeling is the task of fitting a model to a corpus, which can be domai
 transformer-based models are trained using a variant of language modeling, e.g. BERT with masked language modeling,
 GPT-2 with causal language modeling.

-Language modeling can be useful outside of pretraining as well, for example to shift the model distribution to be
+Language modeling can be useful outside of pre-training as well, for example to shift the model distribution to be
 domain-specific: using a language model trained over a very large corpus, and then fine-tuning it to a news dataset or
 on scientific papers e.g. `LysandreJik/arxiv-nlp <https://huggingface.co/lysandre/arxiv-nlp>`__.

@@ -384,6 +384,8 @@ def generic_train(
        train_params["distributed_backend"] = "ddp"

    train_params["accumulate_grad_batches"] = args.accumulate_grad_batches
+    train_params["accelerator"] = extra_train_kwargs.get("accelerator", None)
+    train_params["profiler"] = extra_train_kwargs.get("profiler", None)

    trainer = pl.Trainer.from_argparse_args(
        args,
@@ -7,8 +7,8 @@ to the retriever to extract relevant context documents. The documents are then p
 Such contextualized inputs are passed to the generator.

 Read more about RAG  at https://arxiv.org/abs/2005.11401.
-# Finetuning

+# Finetuning

 Our finetuning logic is based on scripts from [`examples/seq2seq`](https://github.com/huggingface/transformers/tree/master/examples/seq2seq). We accept training data in the same format as specified there - we expect a directory consisting of 6 text files:
 ```bash
@@ -20,10 +20,10 @@ test.source
 test.target
 ```

-A sample finetuning command (run ` ./examples/rag/finetune.py --help` to list all available options):
+A sample finetuning command (run ` ./examples/rag/finetune_rag.py --help` to list all available options):

 ```bash
-python examples/rag/finetune.py \
+python examples/rag/finetune_rag.py \
    --data_dir $DATA_DIR \
    --output_dir $OUTPUT_DIR \
    --model_name_or_path $MODEL_NAME_OR_PATH \
@@ -45,7 +45,7 @@ python examples/rag/consolidate_rag_checkpoint.py \
    --question_encoder_name_or_path facebook/dpr-question_encoder-single-nq-base \
    --dest path/to/checkpoint
 ```
-You will then be able to pass `path/to/checkpoint` as `model_name_or_path` to the `finetune.py` script.
+You will then be able to pass `path/to/checkpoint` as `model_name_or_path` to the `finetune_rag.py` script.


 # Evaluation
@@ -130,3 +130,29 @@ python examples/rag/eval_rag.py \
    --print_predictions \
    --recalculate \ # adding this parameter will force recalculating predictions even if predictions_path already exists
 ```
+
+# Use your own knowledge source
+
+By default, RAG uses the English Wikipedia as a knowledge source, known as the 'wiki_dpr' dataset.
+With `use_custom_knowledge_dataset.py` you can build your own knowledge source, *e.g.* for RAG.
+
+For instance, if documents are serialized as tab-separated csv files with the columns "title" and "text", one can use `use_own_knowledge_dataset.py` as follows:
+```bash
+python examples/rag/use_own_knowledge_dataset.py \
+    --csv_path path/to/my_csv \
+    --output_dir path/to/my_knowledge_dataset \
+```
+
+The created outputs in `path/to/my_knowledge_dataset` can then be used to finetune RAG as follows:
+```bash
+python examples/rag/finetune_rag.py \
+    --data_dir $DATA_DIR \
+    --output_dir $OUTPUT_DIR \
+    --model_name_or_path $MODEL_NAME_OR_PATH \
+    --model_type rag_sequence \
+    --fp16 \
+    --gpus 8
+    --index_name custom
+    --passages_path path/to/data/my_knowledge_dataset
+    --index_path path/to/my_knowledge_dataset_hnsw_index.faiss
+```
@@ -8,7 +8,7 @@ import torch
 from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
 from pytorch_lightning.utilities import rank_zero_only

-from utils import save_json
+from utils_rag import save_json


 def count_trainable_parameters(model):
@@ -38,7 +38,7 @@ def get_checkpoint_callback(output_dir, metric):
        monitor=f"val_{metric}",
        mode="max",
        save_top_k=3,
-        period=0,  # maybe save a checkpoint every time val is run, not just end of epoch.
+        period=1,  # maybe save a checkpoint every time val is run, not just end of epoch.
    )
    return checkpoint_callback

@@ -40,7 +40,6 @@ class RagPyTorchDistributedRetriever(RagRetriever):
            generator_tokenizer=generator_tokenizer,
            index=index,
        )
-
        self.process_group = None

    def init_retrieval(self, distributed_port: int):
@@ -1,12 +1,10 @@
 """Finetuning script for RAG models. Adapted from examples.seq2seq.finetune.py"""

 import argparse
-import glob
 import logging
 import os
 import sys
 import time
-import warnings
 from collections import defaultdict
 from pathlib import Path
 from typing import Any, Dict, List, Tuple
@@ -15,29 +13,31 @@ import numpy as np
 import pytorch_lightning as pl
 import torch
 import torch.distributed as dist
+from pytorch_lightning.accelerators.ddp_accelerator import DDPAccelerator
+from pytorch_lightning.cluster_environments import TorchElasticEnvironment
 from torch.utils.data import DataLoader

 from transformers import (
    AutoConfig,
    AutoTokenizer,
    BartForConditionalGeneration,
+    BatchEncoding,
    RagConfig,
    RagSequenceForGeneration,
    RagTokenForGeneration,
    RagTokenizer,
    T5ForConditionalGeneration,
-    get_linear_schedule_with_warmup,
 )
 from transformers import logging as transformers_logging


-from callbacks import (  # noqa: E402 # isort:skipq
+from callbacks_rag import (  # noqa: E402 # isort:skipq
    get_checkpoint_callback,
    get_early_stopping_callback,
    Seq2SeqLoggingCallback,
 )
 from distributed_retriever import RagPyTorchDistributedRetriever  # noqa: E402 # isort:skip
-from utils import (  # noqa: E402 # isort:skip
+from utils_rag import (  # noqa: E402 # isort:skip
    calculate_exact_match,
    flatten_list,
    get_git_info,
@@ -67,6 +67,30 @@ class AttrDict(dict):
        self.__dict__ = self


+# In PTL >v1.0, `init_ddp_connection` method in the `LightningModule`
+# is no longer used, and is moved into DDPAccelerator instead.
+# We override DDPAccelerator to add our custom logic for initializing the
+# retriever.
+# https://github.com/PyTorchLightning/pytorch-lightning/blob/master/tests/backends/test_accelerator_connector.py
+
+
+class CustomAccel(DDPAccelerator):
+    def __init__(self, trainer=None, **kwargs):
+        # Trainer is set later.
+        super().__init__(trainer, **kwargs)
+
+    def init_ddp_connection(self, global_rank: int, world_size: int, is_slurm_managing_tasks: bool = True):
+        logger.info("Custom init_ddp_connection.")
+        module = self.trainer.model
+        if self.cluster_environment is None:
+            self.cluster_environment = TorchElasticEnvironment()
+        self.distributed_port = module.hparams.distributed_port
+        os.environ["MASTER_PORT"] = str(self.distributed_port)
+        super().init_ddp_connection(global_rank, world_size, is_slurm_managing_tasks)
+        if module.is_rag_model:
+            module.model.rag.retriever.init_retrieval(self.distributed_port)
+
+
 class GenerativeQAModule(BaseTransformer):
    mode = "generative_qa"
    loss_names = ["loss"]
@@ -91,23 +115,24 @@ class GenerativeQAModule(BaseTransformer):
        config = config_class.from_pretrained(hparams.model_name_or_path)

        # set retriever parameters
-        config.index_name = args.index_name or config.index_name
-        config.passages_path = args.passages_path or config.passages_path
-        config.index_path = args.index_path or config.index_path
+        config.index_name = hparams.index_name or config.index_name
+        config.passages_path = hparams.passages_path or config.passages_path
+        config.index_path = hparams.index_path or config.index_path
+        config.use_dummy_dataset = hparams.use_dummy_dataset

        # set extra_model_params for generator configs and load_model
        extra_model_params = ("encoder_layerdrop", "decoder_layerdrop", "attention_dropout", "dropout")
        if self.is_rag_model:
-            if args.prefix is not None:
-                config.generator.prefix = args.prefix
+            if hparams.prefix is not None:
+                config.generator.prefix = hparams.prefix
            config.label_smoothing = hparams.label_smoothing
            hparams, config.generator = set_extra_model_params(extra_model_params, hparams, config.generator)
            retriever = RagPyTorchDistributedRetriever.from_pretrained(hparams.model_name_or_path, config=config)
            model = self.model_class.from_pretrained(hparams.model_name_or_path, config=config, retriever=retriever)
            prefix = config.question_encoder.prefix
        else:
-            if args.prefix is not None:
-                config.prefix = args.prefix
+            if hparams.prefix is not None:
+                config.prefix = hparams.prefix
            hparams, config = set_extra_model_params(extra_model_params, hparams, config)
            model = self.model_class.from_pretrained(hparams.model_name_or_path, config=config)
            prefix = config.prefix
@@ -152,11 +177,9 @@ class GenerativeQAModule(BaseTransformer):
        self.num_workers = hparams.num_workers
        self.distributed_port = self.hparams.distributed_port

-    def init_ddp_connection(self, global_rank: int, world_size: int, is_slurm_managing_tasks: bool = True):
-        logger.info("Custom init_ddp_connection.")
-        os.environ["MASTER_PORT"] = str(self.distributed_port)
-        super().init_ddp_connection(global_rank, world_size, is_slurm_managing_tasks)
-        if self.is_rag_model:
+        # For single GPU training, init_ddp_connection is not called.
+        # So we need to initialize the retrievers here.
+        if hparams.gpus <= 1:
            self.model.retriever.init_retrieval(self.distributed_port)

    def forward(self, input_ids, **kwargs):
@@ -270,6 +293,7 @@ class GenerativeQAModule(BaseTransformer):

    def _generative_step(self, batch: dict) -> dict:
        start_time = time.time()
+        batch = BatchEncoding(batch).to(device=self.model.device)
        generated_ids = self.model.generate(
            batch["input_ids"],
            attention_mask=batch["attention_mask"],
@@ -322,17 +346,6 @@ class GenerativeQAModule(BaseTransformer):

    def train_dataloader(self) -> DataLoader:
        dataloader = self.get_dataloader("train", batch_size=self.hparams.train_batch_size, shuffle=True)
-        t_total = (
-            (len(dataloader.dataset) // (self.hparams.train_batch_size * max(1, self.hparams.gpus)))
-            // self.hparams.accumulate_grad_batches
-            * float(self.hparams.max_epochs)
-        )
-        scheduler = get_linear_schedule_with_warmup(
-            self.opt, num_warmup_steps=self.hparams.warmup_steps, num_training_steps=t_total
-        )
-        if max(scheduler.get_last_lr()) > 0:
-            warnings.warn("All learning rates are 0")
-        self.lr_scheduler = scheduler
        return dataloader

    def val_dataloader(self) -> DataLoader:
@@ -429,10 +442,24 @@ class GenerativeQAModule(BaseTransformer):
            default=None,
            help="Path to the faiss index for custom index. More info about custom indexes in the RagRetriever documentation as well as in `examples/rag/use_own_knowledge_dataset.py`",
        )
+        parser.add_argument(
+            "--use_dummy_dataset",
+            type=bool,
+            default=False,
+            help="Whether to use the dummy version of the dataset index. More info about custom indexes in the RagRetriever documentation as well as in `examples/rag/use_own_knowledge_dataset.py`",
+        )
        return parser


-def main(args, model=None) -> GenerativeQAModule:
+def main(args=None, model=None) -> GenerativeQAModule:
+
+    parser = argparse.ArgumentParser()
+    parser = pl.Trainer.add_argparse_args(parser)
+    parser = GenerativeQAModule.add_model_specific_args(parser, os.getcwd())
+    parser = GenerativeQAModule.add_retriever_specific_args(parser)
+
+    args = args or parser.parse_args()
+
    Path(args.output_dir).mkdir(exist_ok=True)
    if model is None:
        model: GenerativeQAModule = GenerativeQAModule(args)
@@ -461,6 +488,7 @@ def main(args, model=None) -> GenerativeQAModule:
        if args.early_stopping_patience >= 0
        else False
    )
+
    trainer: pl.Trainer = generic_train(
        model,
        args,
@@ -468,31 +496,17 @@ def main(args, model=None) -> GenerativeQAModule:
        checkpoint_callback=get_checkpoint_callback(args.output_dir, model.val_metric),
        early_stopping_callback=es_callback,
        logger=logger,
+        accelerator=CustomAccel() if args.gpus > 1 else None,
    )
    pickle_save(model.hparams, model.output_dir / "hparams.pkl")

    if not args.do_predict:
        return model

-    model.hparams.test_checkpoint = ""
-    checkpoints = list(sorted(glob.glob(os.path.join(args.output_dir, "*.ckpt"), recursive=True)))
-    if checkpoints:
-        model.hparams.test_checkpoint = checkpoints[-1]
-        trainer.resume_from_checkpoint = checkpoints[-1]  # best checkpoint
-    trainer.logger.log_hyperparams(model.hparams)
-
    # test() without a model tests using the best checkpoint automatically
    trainer.test()
-
    return model


 if __name__ == "__main__":
-    parser = argparse.ArgumentParser()
-    parser = pl.Trainer.add_argparse_args(parser)
-    parser = GenerativeQAModule.add_model_specific_args(parser, os.getcwd())
-    parser = GenerativeQAModule.add_retriever_specific_args(parser)
-
-    args = parser.parse_args()
-
-    main(args)
+    main()
@@ -4,7 +4,7 @@ export PYTHONPATH="../":"${PYTHONPATH}"
 # A sample finetuning run, you need to specify data_dir, output_dir and model_name_or_path
 # run ./examples/rag/finetune.sh --help to see all the possible options

-python examples/rag/finetune.py \
+python examples/rag/finetune_rag.py \
    --data_dir $DATA_DIR \
    --output_dir $OUTPUT_DIR \
    --model_name_or_path $MODEL_NAME_OR_PATH \
@@ -0,0 +1,96 @@
+import json
+import logging
+import os
+import sys
+from pathlib import Path
+
+import finetune_rag
+from transformers.file_utils import is_apex_available
+from transformers.testing_utils import (
+    TestCasePlus,
+    execute_subprocess_async,
+    require_torch_gpu,
+    require_torch_multi_gpu,
+)
+
+
+logging.basicConfig(level=logging.DEBUG)
+logger = logging.getLogger()
+
+
+class RagFinetuneExampleTests(TestCasePlus):
+    def _create_dummy_data(self, data_dir):
+        os.makedirs(data_dir, exist_ok=True)
+        contents = {"source": "What is love ?", "target": "life"}
+        n_lines = {"train": 12, "val": 2, "test": 2}
+        for split in ["train", "test", "val"]:
+            for field in ["source", "target"]:
+                content = "\n".join([contents[field]] * n_lines[split])
+                with open(os.path.join(data_dir, f"{split}.{field}"), "w") as f:
+                    f.write(content)
+
+    def _run_finetune(self, gpus: int):
+        stream_handler = logging.StreamHandler(sys.stdout)
+        logger.addHandler(stream_handler)
+
+        tmp_dir = self.get_auto_remove_tmp_dir()
+        output_dir = os.path.join(tmp_dir, "output")
+        data_dir = os.path.join(tmp_dir, "data")
+        self._create_dummy_data(data_dir=data_dir)
+
+        testargs = f"""
+                --data_dir {data_dir} \
+                --output_dir {output_dir} \
+                --model_name_or_path facebook/rag-sequence-base \
+                --model_type rag_sequence \
+                --do_train \
+                --do_predict \
+                --n_val -1 \
+                --val_check_interval 1.0 \
+                --train_batch_size 2 \
+                --eval_batch_size 1 \
+                --max_source_length 25 \
+                --max_target_length 25 \
+                --val_max_target_length 25 \
+                --test_max_target_length 25 \
+                --label_smoothing 0.1 \
+                --dropout 0.1 \
+                --attention_dropout 0.1 \
+                --weight_decay 0.001 \
+                --adam_epsilon 1e-08 \
+                --max_grad_norm 0.1 \
+                --lr_scheduler polynomial \
+                --learning_rate 3e-04 \
+                --num_train_epochs 1 \
+                --warmup_steps 4 \
+                --gradient_accumulation_steps 1 \
+                --distributed-port 8787 \
+                --use_dummy_dataset 1 \
+            """.split()
+
+        if gpus > 0:
+            testargs.append(f"--gpus={gpus}")
+            if is_apex_available():
+                testargs.append("--fp16")
+        else:
+            testargs.append("--gpus=0")
+            testargs.append("--distributed_backend=ddp_cpu")
+            testargs.append("--num_processes=2")
+
+        cmd = [sys.executable, str(Path(finetune_rag.__file__).resolve())] + testargs
+        execute_subprocess_async(cmd, env=self.get_env())
+
+        metrics_save_path = os.path.join(output_dir, "metrics.json")
+        with open(metrics_save_path) as f:
+            result = json.load(f)
+        return result
+
+    @require_torch_gpu
+    def test_finetune_gpu(self):
+        result = self._run_finetune(gpus=1)
+        self.assertGreaterEqual(result["test"][0]["test_avg_em"], 0.2)
+
+    @require_torch_multi_gpu
+    def test_finetune_multigpu(self):
+        result = self._run_finetune(gpus=2)
+        self.assertGreaterEqual(result["test"][0]["test_avg_em"], 0.2)
@@ -7,7 +7,7 @@ from tempfile import TemporaryDirectory
 from typing import List, Optional

 import torch
-from datasets import load_dataset
+from datasets import Features, Sequence, Value, load_dataset

 import faiss
 from transformers import (
@@ -82,10 +82,14 @@ def main(
    # And compute the embeddings
    ctx_encoder = DPRContextEncoder.from_pretrained(rag_example_args.dpr_ctx_encoder_model_name).to(device=device)
    ctx_tokenizer = DPRContextEncoderTokenizerFast.from_pretrained(rag_example_args.dpr_ctx_encoder_model_name)
+    new_features = Features(
+        {"text": Value("string"), "title": Value("string"), "embeddings": Sequence(Value("float32"))}
+    )  # optional, save as float32 instead of float64 to save space
    dataset = dataset.map(
        partial(embed, ctx_encoder=ctx_encoder, ctx_tokenizer=ctx_tokenizer),
        batched=True,
        batch_size=processing_args.batch_size,
+        features=new_features,
    )

    # And finally save your dataset
@@ -13,7 +13,7 @@ streamlit
 elasticsearch
 nltk
 pandas
-datasets
+datasets >= 1.1.3
 fire
 pytest
 conllu
@@ -3,8 +3,7 @@
 python finetune_trainer.py \
    --learning_rate=3e-5 \
    --fp16 \
-    --do_train --do_eval --do_predict \
-    --evaluation_strategy steps \
+    --do_train --do_eval --do_predict --evaluate_during_training \
    --predict_with_generate \
    --n_val 1000 \
    "$@"
@@ -5,8 +5,7 @@ export TPU_NUM_CORES=8
 python xla_spawn.py --num_cores $TPU_NUM_CORES \
    finetune_trainer.py \
    --learning_rate=3e-5 \
-    --do_train --do_eval \
-    --evaluation_strategy steps \
+    --do_train --do_eval --evaluate_during_training \
    --prediction_loss_only \
    --n_val 1000 \
    "$@"
@@ -16,8 +16,7 @@ python finetune_trainer.py \
    --num_train_epochs=6 \
    --save_steps 3000 --eval_steps 3000 \
    --max_source_length $MAX_LEN --max_target_length $MAX_LEN --val_max_target_length $MAX_LEN --test_max_target_length $MAX_LEN \
-    --do_train --do_eval --do_predict \
-    --evaluation_strategy steps \
+    --do_train --do_eval --do_predict --evaluate_during_training\
    --predict_with_generate --logging_first_step \
    --task translation --label_smoothing 0.1 \
    "$@"
@@ -17,8 +17,7 @@ python xla_spawn.py --num_cores $TPU_NUM_CORES \
    --save_steps 500 --eval_steps 500 \
    --logging_first_step --logging_steps 200 \
    --max_source_length $MAX_LEN --max_target_length $MAX_LEN --val_max_target_length $MAX_LEN --test_max_target_length $MAX_LEN \
-    --do_train --do_eval \
-    --evaluation_strategy steps \
+    --do_train --do_eval --evaluate_during_training \
    --prediction_loss_only \
    --task translation --label_smoothing 0.1 \
    "$@"
@@ -19,7 +19,6 @@ python finetune_trainer.py \
    --save_steps 3000 --eval_steps 3000 \
    --logging_first_step \
    --max_target_length 56 --val_max_target_length $MAX_TGT_LEN --test_max_target_length $MAX_TGT_LEN \
-    --do_train --do_eval --do_predict \
-    --evaluation_strategy steps \
+    --do_train --do_eval --do_predict --evaluate_during_training \
    --predict_with_generate --sortish_sampler \
    "$@"
@@ -15,8 +15,7 @@ python finetune_trainer.py \
    --sortish_sampler \
    --num_train_epochs 6 \
    --save_steps 25000 --eval_steps 25000 --logging_steps 1000 \
-    --do_train --do_eval --do_predict \
-    --evaluation_strategy steps \
-    --predict_with_generate --logging_first_step \
+    --do_train --do_eval --do_predict --evaluate_during_training \
+    --predict_with_generate --logging_first_step 
    --task translation \
    "$@"
@@ -1,5 +1,4 @@
 import logging
-import os
 from pathlib import Path

 import numpy as np
@@ -98,7 +97,8 @@ def get_checkpoint_callback(output_dir, metric, save_top_k=1, lower_is_better=Fa
        )

    checkpoint_callback = ModelCheckpoint(
-        filepath=os.path.join(output_dir, exp),
+        dirpath=output_dir,
+        filename=exp,
        monitor=f"val_{metric}",
        mode="min" if "loss" in metric else "max",
        save_top_k=save_top_k,
@@ -189,7 +189,7 @@ class Seq2SeqTrainer(Trainer):
        }

        if self.args.predict_with_generate and not self.args.prediction_loss_only:
-            generated_tokens = model.generate(
+            generated_tokens = self.model.generate(
                inputs["input_ids"],
                attention_mask=inputs["attention_mask"],
                **gen_kwargs,
@@ -4,13 +4,7 @@ from unittest.mock import patch

 from transformers import BertTokenizer, EncoderDecoderModel
 from transformers.file_utils import is_datasets_available
-from transformers.testing_utils import (
-    TestCasePlus,
-    execute_subprocess_async,
-    get_gpu_count,
-    require_torch_non_multi_gpu_but_fix_me,
-    slow,
-)
+from transformers.testing_utils import TestCasePlus, execute_subprocess_async, get_gpu_count, slow
 from transformers.trainer_callback import TrainerState
 from transformers.trainer_utils import set_seed

@@ -52,7 +46,6 @@ class TestFinetuneTrainer(TestCasePlus):
        assert "test_results.json" in contents

    @slow
-    @require_torch_non_multi_gpu_but_fix_me
    def test_finetune_bert2bert(self):
        if not is_datasets_available():
            return
@@ -15,7 +15,8 @@
 """
 Fine-tuning the library models for token classification.
 """
-# You can also adapt this script on your own token classification task and datasets. Pointers for this are left as comments.
+# You can also adapt this script on your own token classification task and datasets. Pointers for this are left as
+# comments.

 import logging
 import os
@@ -24,7 +25,7 @@ from dataclasses import dataclass, field
 from typing import Optional

 import numpy as np
-from datasets import load_dataset
+from datasets import ClassLabel, load_dataset
 from seqeval.metrics import accuracy_score, f1_score, precision_score, recall_score

 import transformers
@@ -198,12 +199,17 @@ def main():

    if training_args.do_train:
        column_names = datasets["train"].column_names
+        features = datasets["train"].features
    else:
        column_names = datasets["validation"].column_names
-    text_column_name = "words" if "words" in column_names else column_names[0]
-    label_column_name = data_args.task_name if data_args.task_name in column_names else column_names[1]
+        features = datasets["validation"].features
+    text_column_name = "tokens" if "tokens" in column_names else column_names[0]
+    label_column_name = (
+        f"{data_args.task_name}_tags" if f"{data_args.task_name}_tags" in column_names else column_names[1]
+    )

-    # Labeling (this part will be easier when https://github.com/huggingface/datasets/issues/797 is solved)
+    # In the event the labels are not a `Sequence[ClassLabel]`, we will need to go through the dataset to get the
+    # unique labels.
    def get_label_list(labels):
        unique_labels = set()
        for label in labels:
@@ -212,8 +218,13 @@ def main():
        label_list.sort()
        return label_list

-    label_list = get_label_list(datasets["train"][label_column_name])
-    label_to_id = {l: i for i, l in enumerate(label_list)}
+    if isinstance(features[label_column_name].feature, ClassLabel):
+        label_list = features[label_column_name].feature.names
+        # No need to convert the labels since they are already ints.
+        label_to_id = {i: i for i in range(len(label_list))}
+    else:
+        label_list = get_label_list(datasets["train"][label_column_name])
+        label_to_id = {l: i for i, l in enumerate(label_list)}
    num_labels = len(label_list)

    # Load pretrained model and tokenizer
@@ -4,6 +4,16 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
+- text: "Paris est la [MASK] de la France."
+- text: "Paris est la capitale de la [MASK]."
+- text: "L'élection américaine a eu [MASK] en novembre 2020."
+- text: "تقع سويسرا في [MASK] أوروبا"
+- text: "إسمي محمد وأسكن في [MASK]."
 ---

 # bert-base-15lang-cased
@@ -33,6 +43,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-15lang-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,10 @@ language: ar
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "تقع سويسرا في [MASK] أوروبا"
+- text: "إسمي محمد وأسكن في [MASK]."
 ---

 # bert-base-ar-cased
@@ -25,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-ar-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-bg-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-de-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-el-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,13 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
+- text: "تقع سويسرا في [MASK] أوروبا"
+- text: "إسمي محمد وأسكن في [MASK]."
 ---

 # bert-base-en-ar-cased
@@ -24,6 +31,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-ar-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-bg-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-bg-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: en
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-de-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-de-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-el-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-el-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-es-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-es-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,14 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
+- text: "Paris est la [MASK] de la France."
+- text: "Paris est la capitale de la [MASK]."
+- text: "L'élection américaine a eu [MASK] en novembre 2020."
 ---

 # bert-base-en-fr-cased
@@ -24,6 +32,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-fr-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-hi-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-hi-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-ru-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-ru-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-sw-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-sw-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-th-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-th-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-tr-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-tr-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-ur-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-ur-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-vi-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-vi-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: multilingual
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Google generated 46 billion [MASK] in revenue."
+- text: "Paris is the capital of [MASK]."
+- text: "Algiers is the largest city in [MASK]."
 ---

 # bert-base-en-zh-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-en-zh-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-es-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -4,6 +4,11 @@ language: fr
 datasets: wikipedia

 license: apache-2.0
+
+widget:
+- text: "Paris est la [MASK] de la France."
+- text: "Paris est la capitale de la [MASK]."
+- text: "L'élection américaine a eu [MASK] en novembre 2020."
 ---

 # bert-base-fr-cased
@@ -24,6 +29,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-fr-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-hi-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-ru-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-sw-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-th-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-tr-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-ur-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-vi-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -24,6 +24,8 @@ model = AutoModel.from_pretrained("Geotrend/bert-base-zh-cased")

 ```

+To generate other smaller versions of multilingual transformers please visit [our Github repo](https://github.com/Geotrend-research/smaller-transformers).
+
 ### How to cite

 ```bibtex
@@ -0,0 +1,38 @@
+---
+language: vn
+---
+
+# BERT for Vietnamese is trained on more 20 GB news dataset
+
+Apply for task sentiment analysis on using [AIViVN's comments dataset](https://www.aivivn.com/contests/6)
+
+The model achieved 0.90268 on the public leaderboard, (winner's score is 0.90087)
+Bert4news is used for a toolkit Vietnames(segmentation and Named Entity Recognition) at ViNLPtoolkit(https://github.com/bino282/ViNLP)
+
+***************New Mar 11 , 2020 ***************
+
+**[BERT](https://github.com/google-research/bert)** (from Google Research and the Toyota Technological Institute at Chicago) released with the paper [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805).
+
+We use word sentencepiece, use basic bert tokenization and same config with bert base with lowercase = False.
+
+You can download trained model:
+- [tensorflow](https://drive.google.com/file/d/1X-sRDYf7moS_h61J3L79NkMVGHP-P-k5/view?usp=sharing).
+- [pytorch](https://drive.google.com/file/d/11aFSTpYIurn-oI2XpAmcCTccB_AonMOu/view?usp=sharing).
+
+
+
+Run training with base config
+
+``` bash
+
+python train_pytorch.py \
+  --model_path=bert4news.pytorch \
+  --max_len=200 \
+  --batch_size=16 \
+  --epochs=6 \
+  --lr=2e-5
+
+```
+
+### Contact information
+For personal communication related to this project, please contact Nha Nguyen Van (nha282@gmail.com).
@@ -5,7 +5,7 @@ datasets:
 - wikipedia
 ---

-# BERT multilingual base model (uncased)
+# BERT multilingual base model (cased)

 Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective.
 It was introduced in [this paper](https://arxiv.org/abs/1810.04805) and first released in
@@ -0,0 +1,20 @@
+### How to use
+
+You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
+set a seed for reproducibility:
+
+```python
+>>> from transformers import pipeline, set_seed
+>>> generator = pipeline('text-generation', model='e-tony/gpt2-rnm')
+>>> set_seed(42)
+>>> generator("Rick: I turned myself into a pickle, Morty!\nMorty: ", max_length=50, num_return_sequences=5)
+
+[{'generated_text': "Rick: I turned myself into a pickle, Morty!\nMorty:  I didn't want to have children. It was my fate! I'll pay my mom and dad.\nSnuffles:  Well, at least we"}, 
+ {'generated_text': "Rick: I turned myself into a pickle, Morty!\nMorty:  you know what happened?\n(Steven begins dragging people down the toilet with his hand. As Steven falls) The whole thing starts.\nA man approaches Steven"}, 
+ {'generated_text': "Rick: I turned myself into a pickle, Morty!\nMorty:  Oh wait! And do you remember what I did to you?\nJerry:  Uh, it didn't hurt. It should have hurt a lot since I"}, 
+ {'generated_text': "Rick: I turned myself into a pickle, Morty!\nMorty:  Rick!\nKraven:  Wait! [wary gasp] What the hell are you doing this time?!\nJerry:  Hey, are you"}, 
+ {'generated_text': "Rick: I turned myself into a pickle, Morty!\nMorty:  Uh.\nJerry:  You don't have to put your finger on me today, do you?\nRick:  It's just, what do you"}]
+```
+
+### Training data
+We used the original `gpt2` model and fine-tuned it on [Rick and Morty transcripts](https://rickandmorty.fandom.com/wiki/Category:Transcripts).
@@ -0,0 +1,92 @@
+---
+language: 
+- pt
+tags:
+- ner
+metrics:
+- f1
+- accuracy
+- precision
+- recall
+---
+
+# RiskData Brazilian Portuguese NER
+
+## Model description
+
+This is a finetunned version from [Neuralmind BERTimbau] (https://github.com/neuralmind-ai/portuguese-bert/blob/master/README.md) for Portuguese language.
+
+For more details, please see, (https://github.com/SecexSaudeTCU/noticias_ner).
+
+## Intended uses & limitations
+
+#### How to use
+
+#### Limitations and bias
+
+- The finetunned model was trained on a corpus with around 180 news articles crawled from Google News.  The original project's purpose was to recognize named entities in news 
+related to fraud and corruption, classifying these entities in four classes: PERSON, ORGANIZATION, PUBLIC INSITUITION and LOCAL (PESSOA, ORGANIZAÇÃO, INSTITUIÇÃO PÚBLICA and LOCAL).
+
+## Training data
+
+The training data can be found at (https://github.com/SecexSaudeTCU/noticias_ner/blob/master/dados/labeled_4_labels.jsonl).
+
+
+## Training procedure
+
+
+## Eval results
+
+accuracy: 0.98, 
+precision: 0.86 
+recall: 0.91
+f1: 0.88
+
+
+The score was calculated using this code:
+
+```python
+    def align_predictions(predictions: np.ndarray, label_ids: np.ndarray) -> Tuple[List[int], List[int]]:
+        preds = np.argmax(predictions, axis=2)
+        batch_size, seq_len = preds.shape
+        out_label_list = [[] for _ in range(batch_size)]
+        preds_list = [[] for _ in range(batch_size)]
+
+        for i in range(batch_size):
+            for j in range(seq_len):
+                if label_ids[i, j] != nn.CrossEntropyLoss().ignore_index:
+                    out_label_list[i].append(id2tag[label_ids[i][j]])
+                    preds_list[i].append(id2tag[preds[i][j]])
+
+        return preds_list, out_label_list
+
+    def compute_metrics(p: EvalPrediction) -> Dict:
+        preds_list, out_label_list = align_predictions(p.predictions, p.label_ids)
+        return {
+            "accuracy_score": accuracy_score(out_label_list, preds_list),
+            "precision": precision_score(out_label_list, preds_list),
+            "recall": recall_score(out_label_list, preds_list),
+            "f1": f1_score(out_label_list, preds_list),
+        }
+```
+
+### BibTeX entry and citation info
+
+For further information about BERTimbau language model:
+
+```bibtex
+@inproceedings{souza2020bertimbau,
+    author    = {Souza, F{\'a}bio and Nogueira, Rodrigo and Lotufo, Roberto},
+    title     = {{BERT}imbau: pretrained {BERT} models for {B}razilian {P}ortuguese},
+    booktitle = {9th Brazilian Conference on Intelligent Systems, {BRACIS}, Rio Grande do Sul, Brazil, October 20-23 (to appear)},
+    year      = {2020}
+}
+
+@article{souza2019portuguese,
+    title={Portuguese Named Entity Recognition using BERT-CRF},
+    author={Souza, F{\'a}bio and Nogueira, Rodrigo and Lotufo, Roberto},
+    journal={arXiv preprint arXiv:1909.10649},
+    url={http://arxiv.org/abs/1909.10649},
+    year={2019}
+}
+```
@@ -0,0 +1,61 @@
+---
+language:
+- bn
+datasets:
+- socian
+- bangla-sentiment-benchmark
+license: mit
+tags:
+- bengali
+- bengali-sentiment
+- sentiment-analysis
+---
+
+# bangla-bert-sentiment
+`bangla-bert-sentiment` is a pretrained model for bengali **Sentiment Analysis** using [bangla-bert-base](https://huggingface.co/sagorsarker/bangla-bert-base) model.
+
+## Datasets Details
+This model was trained with two combined datasets
+* [socian sentiment data](https://github.com/socian-ai/socian-bangla-sentiment-dataset-labeled)
+* [bangla classification dataset](https://github.com/rezacsedu/Classification_Benchmarks_Benglai_NLP)
+
+|||
+|--|--|
+|Data Size| 10889 |
+|Positive| 4999 |
+|Negative| 5890 |
+|Train | 8711 |
+| Test | 2178 |
+
+## Training Details
+Model trained with [simpletransformers](https://github.com/ThilinaRajapakse/simpletransformers) binary classification script with total of **3 epochs** in `google colab gpu`.
+
+
+## Evaluation Details
+Model evaluate with 2178 sentences
+
+Here is the evaluation result details in table
+
+
+|Eval Loss | TP | TN | FP | FN | F1 Score |
+| -------- | -- | -- | -- | -- | -------- |
+| 0.3289 | 880 | 1158 | 59 | 81 | 92.63 |
+
+## Usage
+
+Calculate sentiment from given sentence
+
+```py
+
+from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline
+
+tokenizer = AutoTokenizer.from_pretrained("sagorsarker/bangla-bert-sentiment")
+
+model = AutoModelForSequenceClassification.from_pretrained("sagorsarker/bangla-bert-sentiment")
+
+nlp = pipeline('sentiment-analysis', model=model, tokenizer=tokenizer)
+sentence = "বাংলার ঘরে ঘরে আজ নবান্নের উৎসব"
+nlp(sentence)
+
+```
+
@@ -87,7 +87,7 @@
   "outputs": [],
   "source": [
    "!pip install transformers\n",
-    "!pip install tensorflow==2.1.0"
+    "!pip install --upgrade tensorflow"
   ]
  },
  {
@@ -559,4 +559,4 @@
 },
 "nbformat": 4,
 "nbformat_minor": 4
-}
+}
@@ -119,7 +119,7 @@ extras["dev"] = extras["all"] + extras["testing"] + extras["quality"] + extras["

 setup(
    name="transformers",
-    version="4.0.1",
+    version="4.0.0-rc-1",
    author="Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Sam Shleifer, Patrick von Platen, Sylvain Gugger, Google AI Language Team Authors, Open AI team Authors, Facebook AI Authors, Carnegie Mellon University Authors",
    author_email="thomas@huggingface.co",
    description="State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch",
@@ -2,7 +2,7 @@
 # There's no way to ignore "F401 '...' imported but unused" warnings in this
 # module, but to preserve other warnings. So, don't check this module at all.

-__version__ = "4.0.1"
+__version__ = "4.0.0-rc-1"

 # Work around to update TensorFlow's absl.logging threshold which alters the
 # default Python logging output behavior when present.
@@ -55,6 +55,7 @@ from .data import (
 # Files and general utilities
 from .file_utils import (
    CONFIG_NAME,
+    TOKENIZER_CONFIG_NAME,
    MODEL_CARD_NAME,
    PYTORCH_PRETRAINED_BERT_CACHE,
    PYTORCH_TRANSFORMERS_CACHE,
@@ -98,32 +99,34 @@ from .models.albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfig
 from .models.auto import (
    ALL_PRETRAINED_CONFIG_ARCHIVE_MAP,
    CONFIG_MAPPING,
-    MODEL_NAMES_MAPPING,
    TOKENIZER_MAPPING,
    AutoConfig,
    AutoTokenizer,
 )
-from .models.bart import BartConfig, BartTokenizer
+from .models.bart import BartConfig, BartTokenizer, BART_PRETRAINED_TOKENIZER_ARCHIVE_LIST
 from .models.bert import (
    BERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
+    BERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST,
    BasicTokenizer,
    BertConfig,
    BertTokenizer,
    WordpieceTokenizer,
 )
 from .models.bert_generation import BertGenerationConfig
-from .models.bert_japanese import BertJapaneseTokenizer, CharacterTokenizer, MecabTokenizer
-from .models.bertweet import BertweetTokenizer
+from .models.bert_japanese import BertJapaneseTokenizer, CharacterTokenizer, MecabTokenizer, BERT_JAPANESE_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+from .models.bertweet import BertweetTokenizer, BERT_TWEET_PRETRAINED_TOKENIZER_ARCHIVE_LIST
 from .models.blenderbot import (
    BLENDERBOT_PRETRAINED_CONFIG_ARCHIVE_MAP,
+    BLENDERBOT_LARGE_PRETRAINED_TOKENIZER_ARCHIVE_LIST,
+    BLENDERBOT_SMALL_PRETRAINED_TOKENIZER_ARCHIVE_LIST,
    BlenderbotConfig,
    BlenderbotSmallTokenizer,
    BlenderbotTokenizer,
 )
 from .models.camembert import CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, CamembertConfig
-from .models.ctrl import CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP, CTRLConfig, CTRLTokenizer
-from .models.deberta import DEBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, DebertaConfig, DebertaTokenizer
-from .models.distilbert import DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, DistilBertConfig, DistilBertTokenizer
+from .models.ctrl import CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP, CTRLConfig, CTRLTokenizer, CTRL_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+from .models.deberta import DEBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, DebertaConfig, DebertaTokenizer, DEBERTA_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+from .models.distilbert import DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, DistilBertConfig, DistilBertTokenizer, DISTILBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
 from .models.dpr import (
    DPR_PRETRAINED_CONFIG_ARCHIVE_MAP,
    DPRConfig,
@@ -131,23 +134,26 @@ from .models.dpr import (
    DPRQuestionEncoderTokenizer,
    DPRReaderOutput,
    DPRReaderTokenizer,
+        DPR_CONTEXT_ENCODER_PRETRAINED_TOKENIZER_ARCHIVE_LIST,
+        DPR_QUESTION_ENCODER_PRETRAINED_TOKENIZER_ARCHIVE_LIST,
+        DPR_READER_PRETRAINED_TOKENIZER_ARCHIVE_LIST
 )
-from .models.electra import ELECTRA_PRETRAINED_CONFIG_ARCHIVE_MAP, ElectraConfig, ElectraTokenizer
+from .models.electra import ELECTRA_PRETRAINED_CONFIG_ARCHIVE_MAP, ElectraConfig, ElectraTokenizer, ELECTRA_PRETRAINED_TOKENIZER_ARCHIVE_LIST
 from .models.encoder_decoder import EncoderDecoderConfig
-from .models.flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig, FlaubertTokenizer
-from .models.fsmt import FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP, FSMTConfig, FSMTTokenizer
-from .models.funnel import FUNNEL_PRETRAINED_CONFIG_ARCHIVE_MAP, FunnelConfig, FunnelTokenizer
-from .models.gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config, GPT2Tokenizer
-from .models.herbert import HerbertTokenizer
-from .models.layoutlm import LAYOUTLM_PRETRAINED_CONFIG_ARCHIVE_MAP, LayoutLMConfig, LayoutLMTokenizer
-from .models.longformer import LONGFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP, LongformerConfig, LongformerTokenizer
-from .models.lxmert import LXMERT_PRETRAINED_CONFIG_ARCHIVE_MAP, LxmertConfig, LxmertTokenizer
+from .models.flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig, FlaubertTokenizer, FLAUBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+from .models.fsmt import FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP, FSMTConfig, FSMTTokenizer, FSMT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+from .models.funnel import FUNNEL_PRETRAINED_CONFIG_ARCHIVE_MAP, FunnelConfig, FunnelTokenizer, FUNNEL_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+from .models.gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config, GPT2Tokenizer, GPT2_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+from .models.herbert import HerbertTokenizer, HERBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+from .models.layoutlm import LAYOUTLM_PRETRAINED_CONFIG_ARCHIVE_MAP, LayoutLMConfig, LayoutLMTokenizer, LAYOUTLM_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+from .models.longformer import LONGFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP, LongformerConfig, LongformerTokenizer, LONGFORMER_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+from .models.lxmert import LXMERT_PRETRAINED_CONFIG_ARCHIVE_MAP, LxmertConfig, LxmertTokenizer, LXMERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
 from .models.marian import MarianConfig
 from .models.mbart import MBartConfig
 from .models.mmbt import MMBTConfig
-from .models.mobilebert import MOBILEBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, MobileBertConfig, MobileBertTokenizer
+from .models.mobilebert import MOBILEBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, MobileBertConfig, MobileBertTokenizer, MOBILEBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
 from .models.mt5 import MT5Config
-from .models.openai import OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP, OpenAIGPTConfig, OpenAIGPTTokenizer
+from .models.openai import OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP, OpenAIGPTConfig, OpenAIGPTTokenizer, OPENAIGPT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
 from .models.pegasus import PegasusConfig
 from .models.phobert import PhobertTokenizer
 from .models.prophetnet import PROPHETNET_PRETRAINED_CONFIG_ARCHIVE_MAP, ProphetNetConfig, ProphetNetTokenizer
@@ -205,11 +211,11 @@ from .tokenization_utils_base import (


 if is_sentencepiece_available():
-    from .models.albert import AlbertTokenizer
-    from .models.bert_generation import BertGenerationTokenizer
-    from .models.camembert import CamembertTokenizer
-    from .models.marian import MarianTokenizer
-    from .models.mbart import MBartTokenizer
+    from .models.albert import AlbertTokenizer, ALBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.bert_generation import BertGenerationTokenizer, BERT_GENERATION_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.camembert import CamembertTokenizer, CAMEMBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.marian import MarianTokenizer, MARIAN_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.mbart import MBartTokenizer, MBART_PRETRAINED_TOKENIZER_ARCHIVE_LIST
    from .models.pegasus import PegasusTokenizer
    from .models.reformer import ReformerTokenizer
    from .models.t5 import T5Tokenizer
@@ -220,22 +226,25 @@ else:
    from .utils.dummy_sentencepiece_objects import *

 if is_tokenizers_available():
-    from .models.albert import AlbertTokenizerFast
-    from .models.bart import BartTokenizerFast
-    from .models.bert import BertTokenizerFast
-    from .models.camembert import CamembertTokenizerFast
-    from .models.distilbert import DistilBertTokenizerFast
-    from .models.dpr import DPRContextEncoderTokenizerFast, DPRQuestionEncoderTokenizerFast, DPRReaderTokenizerFast
-    from .models.electra import ElectraTokenizerFast
-    from .models.funnel import FunnelTokenizerFast
-    from .models.gpt2 import GPT2TokenizerFast
-    from .models.herbert import HerbertTokenizerFast
-    from .models.layoutlm import LayoutLMTokenizerFast
-    from .models.longformer import LongformerTokenizerFast
-    from .models.lxmert import LxmertTokenizerFast
-    from .models.mbart import MBartTokenizerFast
-    from .models.mobilebert import MobileBertTokenizerFast
-    from .models.openai import OpenAIGPTTokenizerFast
+    from .models.albert import AlbertTokenizerFast, ALBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.bart import BartTokenizerFast, BART_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.bert import BertTokenizerFast, BERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.camembert import CamembertTokenizerFast, CAMEMBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.distilbert import DistilBertTokenizerFast, DISTILBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.dpr import (DPRContextEncoderTokenizerFast, DPRQuestionEncoderTokenizerFast, DPRReaderTokenizerFast,        DPR_CONTEXT_ENCODER_PRETRAINED_TOKENIZER_ARCHIVE_LIST,
+        DPR_QUESTION_ENCODER_PRETRAINED_TOKENIZER_ARCHIVE_LIST,
+        DPR_READER_PRETRAINED_TOKENIZER_ARCHIVE_LIST)
+
+    from .models.electra import ElectraTokenizerFast, ELECTRA_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.funnel import FunnelTokenizerFast, FUNNEL_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.gpt2 import GPT2TokenizerFast, GPT2_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.herbert import HerbertTokenizerFast, HERBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.layoutlm import LayoutLMTokenizerFast, LAYOUTLM_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.longformer import LongformerTokenizerFast, LONGFORMER_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.lxmert import LxmertTokenizerFast, LXMERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.mbart import MBartTokenizerFast, MBART_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.mobilebert import MobileBertTokenizerFast, MOBILEBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
+    from .models.openai import OpenAIGPTTokenizerFast, OPENAIGPT_PRETRAINED_TOKENIZER_ARCHIVE_LIST
    from .models.pegasus import PegasusTokenizerFast
    from .models.reformer import ReformerTokenizerFast
    from .models.retribert import RetriBertTokenizerFast
@@ -877,7 +886,6 @@ else:


 if is_flax_available():
-    from .models.auto import FLAX_MODEL_MAPPING, FlaxAutoModel
    from .models.bert import FlaxBertModel
    from .models.roberta import FlaxRobertaModel
 else:
@@ -55,6 +55,8 @@ class PretrainedConfig(object):
            Whether or not the model should return all hidden-states.
        output_attentions (:obj:`bool`, `optional`, defaults to :obj:`False`):
            Whether or not the model should returns all attentions.
+        use_cache (:obj:`bool`, `optional`, defaults to :obj:`True`):
+            Whether or not the model should return the last key/values attentions (not used by all models).
        return_dict (:obj:`bool`, `optional`, defaults to :obj:`True`):
            Whether or not the model should return a :class:`~transformers.file_utils.ModelOutput` instead of a plain
            tuple.
@@ -166,6 +168,7 @@ class PretrainedConfig(object):
        self.return_dict = kwargs.pop("return_dict", True)
        self.output_hidden_states = kwargs.pop("output_hidden_states", False)
        self.output_attentions = kwargs.pop("output_attentions", False)
+        self.use_cache = kwargs.pop("use_cache", True)  # Not used by all models
        self.torchscript = kwargs.pop("torchscript", False)  # Only used by PyTorch models
        self.use_bfloat16 = kwargs.pop("use_bfloat16", False)
        self.pruned_heads = kwargs.pop("pruned_heads", {})
@@ -46,7 +46,7 @@ def convert_slow_checkpoint_to_fast(tokenizer_name, checkpoint_name, dump_path,

        add_prefix = True
        if checkpoint_name is None:
-            checkpoint_names = list(tokenizer_class.max_model_input_sizes.keys())
+            raise ValueError("Please supply a checkpoint name")
        else:
            checkpoint_names = [checkpoint_name]

@@ -78,16 +78,16 @@ def convert_slow_checkpoint_to_fast(tokenizer_name, checkpoint_name, dump_path,
                "=> {} with prefix {}, add_prefix {}".format(dump_path_full, checkpoint_prefix_name, add_prefix)
            )

-            if checkpoint in list(tokenizer.pretrained_vocab_files_map.values())[0]:
-                file_path = list(tokenizer.pretrained_vocab_files_map.values())[0][checkpoint]
-                next_char = file_path.split(checkpoint)[-1][0]
-                if next_char == "/":
-                    dump_path_full = os.path.join(dump_path_full, checkpoint_prefix_name)
-                    checkpoint_prefix_name = None
+            # if checkpoint in list(tokenizer.pretrained_vocab_files_map.values())[0]:
+            #     file_path = list(tokenizer.pretrained_vocab_files_map.values())[0][checkpoint]
+            #     next_char = file_path.split(checkpoint)[-1][0]
+            #     if next_char == "/":
+            #         dump_path_full = os.path.join(dump_path_full, checkpoint_prefix_name)
+            #         checkpoint_prefix_name = None

-                logger.info(
-                    "=> {} with prefix {}, add_prefix {}".format(dump_path_full, checkpoint_prefix_name, add_prefix)
-                )
+            #     logger.info(
+            #         "=> {} with prefix {}, add_prefix {}".format(dump_path_full, checkpoint_prefix_name, add_prefix)
+            #     )

            file_names = tokenizer.save_pretrained(
                dump_path_full, legacy_format=False, filename_prefix=checkpoint_prefix_name
@@ -229,7 +229,7 @@ class LineByLineWithSOPTextDataset(Dataset):
        # to `block_size` anyways, so short sequences are generally wasted
        # computation. However, we *sometimes*
        # (i.e., short_seq_prob == 0.1 == 10% of the time) want to use shorter
-        # sequences to minimize the mismatch between pretraining and fine-tuning.
+        # sequences to minimize the mismatch between pre-training and fine-tuning.
        # The `target_seq_length` is just a rough target however, whereas
        # `block_size` is a hard limit.
        target_seq_length = max_num_tokens
@@ -425,7 +425,7 @@ class TextDatasetForNextSentencePrediction(Dataset):
        # to `block_size` anyways, so short sequences are generally wasted
        # computation. However, we *sometimes*
        # (i.e., short_seq_prob == 0.1 == 10% of the time) want to use shorter
-        # sequences to minimize the mismatch between pretraining and fine-tuning.
+        # sequences to minimize the mismatch between pre-training and fine-tuning.
        # The `target_seq_length` is just a rough target however, whereas
        # `block_size` is a hard limit.
        target_seq_length = max_num_tokens
@@ -234,6 +234,7 @@ WEIGHTS_NAME = "pytorch_model.bin"
 TF2_WEIGHTS_NAME = "tf_model.h5"
 TF_WEIGHTS_NAME = "model.ckpt"
 CONFIG_NAME = "config.json"
+TOKENIZER_CONFIG_NAME = "tokenizer_config.json"
 MODEL_CARD_NAME = "modelcard.json"

 SENTENCEPIECE_UNDERLINE = "▁"
@@ -146,13 +146,13 @@ class RepetitionPenaltyLogitsProcessor(LogitsProcessor):
        self.penalty = penalty

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
-        for i in range(scores.shape[0]):
-            for previous_token in set(input_ids[i].tolist()):
-                # if score < 0 then repetition penalty has to be multiplied to reduce the previous token probability
-                if scores[i, previous_token] < 0:
-                    scores[i, previous_token] *= self.penalty
-                else:
-                    scores[i, previous_token] /= self.penalty
+        ranges = torch.arange(scores.shape[0])
+        score = scores[ranges[:, None], input_ids]
+
+        # if score < 0 then repetition penalty has to be multiplied to reduce the previous token probability
+        score = torch.where(score < 0, score * self.penalty, score / self.penalty)
+
+        scores[ranges[:, None], input_ids] = score
        return scores


@@ -38,7 +38,6 @@ class TFGenerationMixin:

    def _use_cache(self, outputs, use_cache):
        """During generation, decide whether to pass the `past` variable to the next forward pass."""
-        use_cache = getattr(self.config, "use_cache", False)
        if len(outputs) <= 1 or use_cache is False:
            return False
        if hasattr(self.config, "mem_len") and self.config.mem_len == 0:
@@ -195,6 +194,7 @@ class TFGenerationMixin:
        min_length = min_length if min_length is not None else self.config.min_length
        do_sample = do_sample if do_sample is not None else self.config.do_sample
        early_stopping = early_stopping if early_stopping is not None else self.config.early_stopping
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
        num_beams = num_beams if num_beams is not None else self.config.num_beams
        temperature = temperature if temperature is not None else self.config.temperature
        top_k = top_k if top_k is not None else self.config.top_k
@@ -224,6 +224,7 @@ class TFGenerationMixin:
        assert isinstance(min_length, int) and min_length >= 0, "`min_length` should be a positive integer."
        assert isinstance(do_sample, bool), "`do_sample` should be a boolean."
        assert isinstance(early_stopping, bool), "`early_stopping` should be a boolean."
+        assert isinstance(use_cache, bool), "`use_cache` should be a boolean."
        assert isinstance(num_beams, int) and num_beams > 0, "`num_beams` should be a strictly positive integer."
        assert temperature > 0, "`temperature` should be strictly positive."
        assert isinstance(top_k, int) and top_k >= 0, "`top_k` should be a positive integer."
@@ -244,12 +244,12 @@ class GenerationMixin:

        # the following idea is largely copied from this PR: https://github.com/huggingface/transformers/pull/5420/files
        # all samplers can be found in `generation_utils_samplers.py`
+        if temperature is not None and temperature != 1.0:
+            warpers.append(TemperatureLogitsWarper(temperature))
        if top_k is not None and top_k != 0:
            warpers.append(TopKLogitsWarper(top_k=top_k, min_tokens_to_keep=(2 if num_beams > 1 else 1)))
        if top_p is not None and top_p < 1.0:
            warpers.append(TopPLogitsWarper(top_p=top_p, min_tokens_to_keep=(2 if num_beams > 1 else 1)))
-        if temperature is not None and temperature != 1.0:
-            warpers.append(TemperatureLogitsWarper(temperature))
        return warpers

    def _get_logits_processor(
@@ -462,6 +462,7 @@ class GenerationMixin:
        pad_token_id = pad_token_id if pad_token_id is not None else self.config.pad_token_id
        bos_token_id = bos_token_id if bos_token_id is not None else self.config.bos_token_id
        eos_token_id = eos_token_id if eos_token_id is not None else self.config.eos_token_id
+        use_cache = use_cache if use_cache is not None else self.config.use_cache

        if input_ids is None:
            # init `input_ids` with bos_token_id
@@ -2,7 +2,6 @@
 import math
 import os

-from .trainer_utils import EvaluationStrategy
 from .utils import logging


@@ -213,13 +212,13 @@ def run_hp_search_ray(trainer, n_trials: int, direction: str, **kwargs) -> BestR
        # Check for `do_eval` and `eval_during_training` for schedulers that require intermediate reporting.
        if isinstance(
            kwargs["scheduler"], (ASHAScheduler, MedianStoppingRule, HyperBandForBOHB, PopulationBasedTraining)
-        ) and (not trainer.args.do_eval or trainer.args.evaluation_strategy == EvaluationStrategy.NO):
+        ) and (not trainer.args.do_eval or not trainer.args.evaluate_during_training):
            raise RuntimeError(
                "You are using {cls} as a scheduler but you haven't enabled evaluation during training. "
                "This means your trials will not report intermediate results to Ray Tune, and "
                "can thus not be stopped early or used to exploit other trials parameters. "
                "If this is what you want, do not use {cls}. If you would like to use {cls}, "
-                "make sure you pass `do_eval=True` and `evaluation_strategy='steps'` in the "
+                "make sure you pass `do_eval=True` and `evaluate_during_training=True` in the "
                "Trainer `args`.".format(cls=type(kwargs["scheduler"]).__name__)
            )

@@ -164,9 +164,9 @@ def load_pytorch_weights_in_tf2_model(tf_model, pt_state_dict, tf_inputs=None, a
            if allow_missing_keys:
                missing_keys.append(name)
                continue
-            elif tf_model._keys_to_ignore_on_load_missing is not None:
+            elif tf_model.authorized_missing_keys is not None:
                # authorized missing keys don't have to be loaded
-                if any(re.search(pat, name) is not None for pat in tf_model._keys_to_ignore_on_load_missing):
+                if any(re.search(pat, name) is not None for pat in tf_model.authorized_missing_keys):
                    continue

            raise AttributeError("{} not found in PyTorch model".format(name))
@@ -209,11 +209,11 @@ def load_pytorch_weights_in_tf2_model(tf_model, pt_state_dict, tf_inputs=None, a

    unexpected_keys = list(all_pytorch_weights)

-    if tf_model._keys_to_ignore_on_load_missing is not None:
-        for pat in tf_model._keys_to_ignore_on_load_missing:
+    if tf_model.authorized_missing_keys is not None:
+        for pat in tf_model.authorized_missing_keys:
            missing_keys = [k for k in missing_keys if re.search(pat, k) is None]
-    if tf_model._keys_to_ignore_on_load_unexpected is not None:
-        for pat in tf_model._keys_to_ignore_on_load_unexpected:
+    if tf_model.authorized_unexpected_keys is not None:
+        for pat in tf_model.authorized_unexpected_keys:
            unexpected_keys = [k for k in unexpected_keys if re.search(pat, k) is None]

    if len(unexpected_keys) > 0:
@@ -343,15 +343,15 @@ class TFPreTrainedModel(tf.keras.Model, TFModelUtilsMixin, TFGenerationMixin):
          :class:`~transformers.PretrainedConfig` to use as configuration class for this model architecture.
        - **base_model_prefix** (:obj:`str`) -- A string indicating the attribute associated to the base model in
          derived classes of the same architecture adding modules on top of the base model.
+        - **authorized_missing_keys** (:obj:`List[str]`, `optional`) -- A list of re pattern of tensor names to ignore
+          from the model when loading the model weights (and avoid unnecessary warnings).
+        - **authorized_unexpected_keys** (:obj:`List[str]`, `optional`) -- A list of re pattern of tensor names to
+          ignore from the weights when loading the model weights (and avoid unnecessary warnings).
    """
    config_class = None
    base_model_prefix = ""
-    # a list of re pattern of tensor names to ignore from the model when loading the model weights
-    # (and avoid unnecessary warnings).
-    _keys_to_ignore_on_load_missing = None
-    # a list of re pattern of tensor names to ignore from the weights when loading the model weights
-    # (and avoid unnecessary warnings).
-    _keys_to_ignore_on_load_unexpected = None
+    authorized_missing_keys = None
+    authorized_unexpected_keys = None

    @property
    def dummy_inputs(self) -> Dict[str, tf.Tensor]:
@@ -742,12 +742,12 @@ class TFPreTrainedModel(tf.keras.Model, TFModelUtilsMixin, TFGenerationMixin):

        model(model.dummy_inputs, training=False)  # Make sure restore ops are run

-        if cls._keys_to_ignore_on_load_missing is not None:
-            for pat in cls._keys_to_ignore_on_load_missing:
+        if cls.authorized_missing_keys is not None:
+            for pat in cls.authorized_missing_keys:
                missing_keys = [k for k in missing_keys if re.search(pat, k) is None]

-        if cls._keys_to_ignore_on_load_unexpected is not None:
-            for pat in cls._keys_to_ignore_on_load_unexpected:
+        if cls.authorized_unexpected_keys is not None:
+            for pat in cls.authorized_unexpected_keys:
                unexpected_keys = [k for k in unexpected_keys if re.search(pat, k) is None]

        if len(unexpected_keys) > 0:
@@ -404,18 +404,17 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin):

        - **base_model_prefix** (:obj:`str`) -- A string indicating the attribute associated to the base model in
          derived classes of the same architecture adding modules on top of the base model.
+        - **authorized_missing_keys** (:obj:`Optional[List[str]]`) -- A list of re pattern of tensor names to ignore
+          when loading the model (and avoid unnecessary warnings).
+        - **keys_to_never_save** (:obj:`Optional[List[str]]`) -- A list of of tensor names to ignore when saving the
+          model (useful for keys that aren't trained, but which are deterministic)
+
    """
    config_class = None
    base_model_prefix = ""
-    # a list of re pattern of tensor names to ignore from the model when loading the model weights
-    # (and avoid unnecessary warnings).
-    _keys_to_ignore_on_load_missing = None
-    # a list of re pattern of tensor names to ignore from the weights when loading the model weights
-    # (and avoid unnecessary warnings).
-    _keys_to_ignore_on_load_unexpected = None
-    # a list of of tensor names to ignore when saving the model (useful for keys that aren't
-    # trained, but which are deterministic)
-    _keys_to_ignore_on_save = None
+    authorized_missing_keys = None
+    authorized_unexpected_keys = None
+    keys_to_never_save = None

    @property
    def dummy_inputs(self) -> Dict[str, torch.Tensor]:
@@ -720,8 +719,8 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin):
        state_dict = model_to_save.state_dict()

        # Handle the case where some state_dict keys shouldn't be saved
-        if self._keys_to_ignore_on_save is not None:
-            state_dict = {k: v for k, v in state_dict.items() if k not in self._keys_to_ignore_on_save}
+        if self.keys_to_never_save is not None:
+            state_dict = {k: v for k, v in state_dict.items() if k not in self.keys_to_never_save}

        # If we save using the predefined names, we can load using `from_pretrained`
        output_model_file = os.path.join(save_directory, WEIGHTS_NAME)
@@ -1035,12 +1034,12 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin):

            # Some models may have keys that are not in the state by design, removing them before needlessly warning
            # the user.
-            if cls._keys_to_ignore_on_load_missing is not None:
-                for pat in cls._keys_to_ignore_on_load_missing:
+            if cls.authorized_missing_keys is not None:
+                for pat in cls.authorized_missing_keys:
                    missing_keys = [k for k in missing_keys if re.search(pat, k) is None]

-            if cls._keys_to_ignore_on_load_unexpected is not None:
-                for pat in cls._keys_to_ignore_on_load_unexpected:
+            if cls.authorized_unexpected_keys is not None:
+                for pat in cls.authorized_unexpected_keys:
                    unexpected_keys = [k for k in unexpected_keys if re.search(pat, k) is None]

            if len(unexpected_keys) > 0:
@@ -7,10 +7,10 @@ from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertCo


 if is_sentencepiece_available():
-    from .tokenization_albert import AlbertTokenizer
+    from .tokenization_albert import AlbertTokenizer, ALBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST

 if is_tokenizers_available():
-    from .tokenization_albert_fast import AlbertTokenizerFast
+    from .tokenization_albert_fast import AlbertTokenizerFast, ALBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST

 if is_torch_available():
    from .modeling_albert import (
@@ -214,7 +214,6 @@ class AlbertEmbeddings(nn.Module):

        # position_ids (1, len position emb) is contiguous in memory and exported when serialized
        self.register_buffer("position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)))
-        self.position_embedding_type = getattr(config, "position_embedding_type", "absolute")

    # Copied from transformers.models.bert.modeling_bert.BertEmbeddings.forward
    def forward(self, input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None):
@@ -266,11 +265,6 @@ class AlbertAttention(nn.Module):
        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.pruned_heads = set()

-        self.position_embedding_type = getattr(config, "position_embedding_type", "absolute")
-        if self.position_embedding_type == "relative_key" or self.position_embedding_type == "relative_key_query":
-            self.max_position_embeddings = config.max_position_embeddings
-            self.distance_embedding = nn.Embedding(2 * config.max_position_embeddings - 1, self.attention_head_size)
-
    # Copied from transformers.models.bert.modeling_bert.BertSelfAttention.transpose_for_scores
    def transpose_for_scores(self, x):
        new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
@@ -465,7 +459,7 @@ class AlbertPreTrainedModel(PreTrainedModel):

    config_class = AlbertConfig
    base_model_prefix = "albert"
-    _keys_to_ignore_on_load_missing = [r"position_ids"]
+    authorized_missing_keys = [r"position_ids"]

    def _init_weights(self, module):
        """Initialize the weights."""
@@ -711,7 +705,7 @@ class AlbertModel(AlbertPreTrainedModel):

@add_start_docstrings(
    """
-    Albert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a
+    Albert Model with two heads on top as done during the pre-training: a `masked language modeling` head and a
    `sentence order prediction (classification)` head.
    """,
    ALBERT_START_DOCSTRING,
@@ -857,7 +851,7 @@ class AlbertSOPHead(nn.Module):
 )
 class AlbertForMaskedLM(AlbertPreTrainedModel):

-    _keys_to_ignore_on_load_unexpected = [r"pooler"]
+    authorized_unexpected_keys = [r"pooler"]

    def __init__(self, config):
        super().__init__(config)
@@ -1027,7 +1021,7 @@ class AlbertForSequenceClassification(AlbertPreTrainedModel):
 )
 class AlbertForTokenClassification(AlbertPreTrainedModel):

-    _keys_to_ignore_on_load_unexpected = [r"pooler"]
+    authorized_unexpected_keys = [r"pooler"]

    def __init__(self, config):
        super().__init__(config)
@@ -1116,7 +1110,7 @@ class AlbertForTokenClassification(AlbertPreTrainedModel):
 )
 class AlbertForQuestionAnswering(AlbertPreTrainedModel):

-    _keys_to_ignore_on_load_unexpected = [r"pooler"]
+    authorized_unexpected_keys = [r"pooler"]

    def __init__(self, config):
        super().__init__(config)
@@ -768,7 +768,7 @@ class TFAlbertModel(TFAlbertPreTrainedModel):

@add_start_docstrings(
    """
-    Albert Model with two heads on top for pretraining: a `masked language modeling` head and a `sentence order
+    Albert Model with two heads on top for pre-training: a `masked language modeling` head and a `sentence order
    prediction` (classification) head.
    """,
    ALBERT_START_DOCSTRING,
@@ -843,7 +843,7 @@ class TFAlbertSOPHead(tf.keras.layers.Layer):
@add_start_docstrings("""Albert Model with a `language modeling` head on top. """, ALBERT_START_DOCSTRING)
 class TFAlbertForMaskedLM(TFAlbertPreTrainedModel, TFMaskedLanguageModelingLoss):

-    _keys_to_ignore_on_load_missing = [r"pooler"]
+    authorized_missing_keys = [r"pooler"]

    def __init__(self, config, *inputs, **kwargs):
        super().__init__(config, *inputs, **kwargs)
@@ -1013,7 +1013,7 @@ class TFAlbertForSequenceClassification(TFAlbertPreTrainedModel, TFSequenceClass
 )
 class TFAlbertForTokenClassification(TFAlbertPreTrainedModel, TFTokenClassificationLoss):

-    _keys_to_ignore_on_load_missing = [r"pooler"]
+    authorized_missing_keys = [r"pooler"]

    def __init__(self, config, *inputs, **kwargs):
        super().__init__(config, *inputs, **kwargs)
@@ -1100,7 +1100,7 @@ class TFAlbertForTokenClassification(TFAlbertPreTrainedModel, TFTokenClassificat
 )
 class TFAlbertForQuestionAnswering(TFAlbertPreTrainedModel, TFQuestionAnsweringLoss):

-    _keys_to_ignore_on_load_missing = [r"pooler"]
+    authorized_missing_keys = [r"pooler"]

    def __init__(self, config, *inputs, **kwargs):
        super().__init__(config, *inputs, **kwargs)
@@ -27,31 +27,20 @@ from ...utils import logging


 logger = logging.get_logger(__name__)
+
 VOCAB_FILES_NAMES = {"vocab_file": "spiece.model"}

-PRETRAINED_VOCAB_FILES_MAP = {
-    "vocab_file": {
-        "albert-base-v1": "https://huggingface.co/albert-base-v1/resolve/main/spiece.model",
-        "albert-large-v1": "https://huggingface.co/albert-large-v1/resolve/main/spiece.model",
-        "albert-xlarge-v1": "https://huggingface.co/albert-xlarge-v1/resolve/main/spiece.model",
-        "albert-xxlarge-v1": "https://huggingface.co/albert-xxlarge-v1/resolve/main/spiece.model",
-        "albert-base-v2": "https://huggingface.co/albert-base-v2/resolve/main/spiece.model",
-        "albert-large-v2": "https://huggingface.co/albert-large-v2/resolve/main/spiece.model",
-        "albert-xlarge-v2": "https://huggingface.co/albert-xlarge-v2/resolve/main/spiece.model",
-        "albert-xxlarge-v2": "https://huggingface.co/albert-xxlarge-v2/resolve/main/spiece.model",
-    }
-}
-
-PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
-    "albert-base-v1": 512,
-    "albert-large-v1": 512,
-    "albert-xlarge-v1": 512,
-    "albert-xxlarge-v1": 512,
-    "albert-base-v2": 512,
-    "albert-large-v2": 512,
-    "albert-xlarge-v2": 512,
-    "albert-xxlarge-v2": 512,
-}
+ALBERT_PRETRAINED_TOKENIZER_ARCHIVE_LIST = [
+    "albert-base-v1",
+    "albert-large-v1",
+    "albert-xlarge-v1",
+    "albert-xxlarge-v1",
+    "albert-base-v2",
+    "albert-large-v2",
+    "albert-xlarge-v2",
+    "albert-xxlarge-v2",
+    # See all ALBERT models at https://huggingface.co/models?filter=albert
+]

 SPIECE_UNDERLINE = "▁"

@@ -108,8 +97,7 @@ class AlbertTokenizer(PreTrainedTokenizer):
    """

    vocab_files_names = VOCAB_FILES_NAMES
-    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
-    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    max_model_input_sizes = 512

    def __init__(
        self,
Author	SHA1	Message	Date
Thomas Wolf	4cec16ea28	mbart - marian - mobilebert - openai	2020-11-25 23:52:09 +01:00
Thomas Wolf	625496b1f7	longformer - lxmert	2020-11-25 19:30:33 +01:00
Thomas Wolf	52bb1bb40e	herbert and layoutlm	2020-11-25 19:14:55 +01:00
Thomas Wolf	d805add061	funnel and gpt2	2020-11-25 19:03:07 +01:00
Thomas Wolf	9b71c262de	flaubert - fsmt	2020-11-25 18:49:18 +01:00
Thomas Wolf	726ded8647	dpr and electra	2020-11-25 18:36:19 +01:00
Thomas Wolf	812f276272	ctrl - deberta - distilbert	2020-11-25 15:40:34 +01:00
Thomas Wolf	20fdaac34b	adding camembert	2020-11-25 13:21:45 +01:00
Thomas Wolf	0c0aefb23b	adding blenderbot - also support in auto	2020-11-25 13:14:17 +01:00
Thomas Wolf	aaf7ba4ec4	bert japanese and bert tweet	2020-11-25 12:42:10 +01:00
Thomas Wolf	824fae28cf	bert_generation	2020-11-25 12:09:05 +01:00
Thomas Wolf	d86b95aa86	clean-up	2020-11-25 00:20:58 +01:00
Thomas Wolf	b40ad98fc5	update bert tokenizer	2020-11-25 00:19:48 +01:00
Thomas Wolf	8c71c1b404	switch bart tokenizer	2020-11-24 18:23:25 +01:00
Thomas Wolf	dd50d6b932	switch Albert tokenizer to common hub access - fix autotokenizer reloading	2020-11-24 16:04:58 +01:00
Thomas Wolf	9876a9e06f	add tokenizer_class_name to tokenizers config	2020-11-24 00:05:26 +01:00
Sylvain Gugger	49759c0cda	Document new training argument	2020-11-23 15:02:59 -05:00
alexorona Sylvain Gugger Lysandre Debut LysandreJik	1cd9be2aeb	gpt2 and t5 parallel modeling (#8696 ) * gpt2 and t5 parallel modeling * model_parallel utils update * adding missing model_parallel_utils Adds missing model_parallel_utils and reverses the changes to code in modeling_gpt2 and modeling_t5 * training_args reformat Reformatted training_args * style formatting Style formatting doc string length on training_args and model_parallel_utils * style changes make style && make quality for training_args and model_parallel_utils. * adding tests * minor change in trainer reverts loss calculation * Update training_args.py * Update training_args.py added back docstring language for adam_beta1 and adam_beta2 * Update trainer.py * Update src/transformers/trainer.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Fix style & rebase Co-authored-by: Lysandre Debut <lysandre@huggingface.co> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Co-authored-by: LysandreJik <lysandre.debut@reseau.eseo.fr>	2020-11-23 14:41:23 -05:00
Stas Bekman	1e45bef0a7	[trainer] make generate work with multigpu (#8716 ) * make generate work with multigpu * better fix - thanks @sgugger	2020-11-23 10:57:27 -08:00
Sylvain GuggerandLysandre Debut	900024273b	Change default cache path (#8734 ) * Change default cache path * Document changes * Apply suggestions from code review Co-authored-by: Lysandre Debut <lysandre@huggingface.co> Co-authored-by: Lysandre Debut <lysandre@huggingface.co>	2020-11-23 13:56:45 -05:00
Julien ChaumondandSylvain Gugger	0cc5ab1333	Improve bert-japanese tokenizer handling (#8659 ) * Make ci fail * Try to make tests actually run? * CI finally failing? * Fix CI * Revert "Fix CI" This reverts commit ca7923be7334d4e571b023478ebdd6b33dfd0ebb. * Ooops wrong one * one more try * Ok ok let's move this elsewhere * Alternative to globals() (#8667) * Alternative to globals() * Error is raised later so return None * Sentencepiece not installed make some tokenizers None * Apply Lysandre wisdom * Slightly clearer comment? cc @sgugger Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>	2020-11-23 11:15:02 -05:00
Amine Abdaoui	eec76615f6	[model_cards]: control input examples of Geotrend models (#8727 ) * [model_cards]: control arabic model examples * [model_cards]: control input examples of Geotrend models * [model_cards]: add link to generatation script	2020-11-23 11:09:50 -05:00
Jessica YungandLysandre Debut	143b564e59	Add pip install update to resolve import error in transformers notebook (#8616 ) * Add pip install update to resolve import error Add pip install upgrade tensorflow-gpu to remove error below: ``` --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-2-094fadb93f3f> in <module>() 1 import torch ----> 2 from transformers import AutoModel, AutoTokenizer, BertTokenizer 3 4 torch.set_grad_enabled(False) 4 frames /usr/local/lib/python3.6/dist-packages/transformers/__init__.py in <module>() 133 134 # Pipelines --> 135 from .pipelines import ( 136 Conversation, 137 ConversationalPipeline, /usr/local/lib/python3.6/dist-packages/transformers/pipelines.py in <module>() 46 import tensorflow as tf 47 ---> 48 from .modeling_tf_auto import ( 49 TF_MODEL_FOR_QUESTION_ANSWERING_MAPPING, 50 TF_MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING, /usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_auto.py in <module>() 49 from .configuration_utils import PretrainedConfig 50 from .file_utils import add_start_docstrings ---> 51 from .modeling_tf_albert import ( 52 TFAlbertForMaskedLM, 53 TFAlbertForMultipleChoice, /usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_albert.py in <module>() 22 import tensorflow as tf 23 ---> 24 from .activations_tf import get_tf_activation 25 from .configuration_albert import AlbertConfig 26 from .file_utils import ( /usr/local/lib/python3.6/dist-packages/transformers/activations_tf.py in <module>() 52 "gelu": tf.keras.layers.Activation(gelu), 53 "relu": tf.keras.activations.relu, ---> 54 "swish": tf.keras.activations.swish, 55 "silu": tf.keras.activations.swish, 56 "gelu_new": tf.keras.layers.Activation(gelu_new), AttributeError: module 'tensorflow_core.python.keras.api._v2.keras.activations' has no attribute 'swish' ``` I have tried running the colab after this change and it seems to work fine (all the cells run with no errors). * Update notebooks/02-transformers.ipynb only need to upgrade tensorflow, not tensorflow-gpu. Co-authored-by: Lysandre Debut <lysandre@huggingface.co> Co-authored-by: Lysandre Debut <lysandre@huggingface.co>	2020-11-23 09:58:52 -05:00
Yossi Synett	18c8cf000b	Fix bug in x-attentions output for roberta and harden test to catch it (#8660 )	2020-11-23 13:28:29 +01:00
Tony	48cc224703	[model_cards] Add card for gpt2-rnm (#8673 )	2020-11-23 05:52:29 -05:00
Nguyen Van NhaandJulien Chaumond	52585e40af	create README.md (#8682 ) * create README.md * Apply suggestions from code review Co-authored-by: Julien Chaumond <chaumond@gmail.com>	2020-11-23 05:51:54 -05:00
Sagor Sarker	b5187e317f	added bangla-bert-sentiment model card (#8687 )	2020-11-23 05:51:16 -05:00
moniquebmandJulien Chaumond	b6d864e2f0	Create README.md (#8630 ) * Create README.md * correct metrics id cc @lhoestq Co-authored-by: Julien Chaumond <chaumond@gmail.com>	2020-11-23 04:48:10 -05:00
Santiago Castro	e1f3156b21	Fix many typos (#8708 )	2020-11-21 22:58:10 -05:00
Patrick von Platen	9c0afdaf7b	fix flaky ci (#8694 )	2020-11-20 22:07:21 +01:00
Binoy Dalal	29bdb88368	Vectorize RepetitionPenaltyLogitsProcessor to improve performance (#8598 ) * refactored exisiting nested loops to vectorized implementation * replaced explicit indexing with torch.where * modifying score for previous input_ids only	2020-11-20 19:59:06 +01:00
Roman Kalyakin	2594bd8b73	moved temperature wrapper before topP/topK (#8686 )	2020-11-20 19:33:54 +01:00
Quentin Lhoest	8062fa63c5	Fix rag finetuning + add finetuning test (#8585 ) * replace init_ddp_connection for index init * style * add finetune test * add test data * move generate tensors to device * add test on EM metric * style * allow multi process test * keep gloo process group for retrieval * add multi-gpu test * use custom accelerator * clean test finetune * minor * style * style * typo * use python call instead of imported main fumction * return_dict fix in modeling_rag * use float32 in retrieval * store as float32 as well in the custom knowledge dataset example * style * rename to finetune_rag * style * update readme * rename utils and callbacks to utils_rag and callbacks_rag * fix test * patrick's comments * generate dummy data in the finetue test script * remove dummy data files * style	2020-11-20 19:05:03 +01:00
Sylvain Gugger	63e91f5fde	Document adam betas TrainingArguments (#8688 )	2020-11-20 09:27:25 -05:00
Kevin Canwen Xu	94caaa93c2	Update the bibtex with EMNLP demo (#8678 ) * Update the bibtex with EMNLP demo * Update README.md * Update README.md	2020-11-20 13:26:33 +08:00
Sylvain Gugger	6494910f27	Add sentencepiece to the CI and fix tests (#8672 ) * Fix the CI and tests * Fix quality * Remove that m form nowhere	2020-11-19 16:44:20 -05:00
Stas Bekman	0ad45e108d	[examples/seq2seq] fix PL deprecation warning (#8577 ) * fix deprecation warning * fix	2020-11-19 21:46:04 +01:00
Arindum Roy	0e19a4c2d6	Update bert-base-multilingual-cased-README.md (#8668 ) The heading was originally uncased, which did not reflect the contents of this README. Changed it to cased.	2020-11-19 15:45:06 -05:00
Stas Bekman	06518404cb	revert	2020-11-19 12:12:46 -08:00
Stas Bekman	297a29382f	Please fix your software not to ping master You may be unaware but you're running some software that meddles with every commit on https://github.com/huggingface/transformers/ Something is wrong with the software you're using. It adds a reference to almost every PR in the master tree. Which is very wrong. Please check your software and please don't do it again. Example: see the bottom of this PR and most other PRs: https://github.com/huggingface/transformers/pull/8639	2020-11-19 12:11:35 -08:00
Stas Bekman	42111f1d56	[tokenizers] convert_to_tensors: don't reconvert when the type is already right (#8283 ) * don't reconvert when the type is already right * better name * adjust logic as suggested * merge	2020-11-19 12:06:01 -08:00
Sylvain Gugger	20b658607e	Fix run_ner script (#8664 ) * Fix run_ner script * Pin datasets	2020-11-19 13:59:30 -05:00
Zhylko Dimaandpatrickvonplaten	ca0109bd68	`disable_ngram_loss` fix for prophetnet (#8554 ) * `disable_ngram_loss` fix for prophetnet * add changes documentation * fix _compute_loss to use mean reduction and -100 to masked tokens & remove unnecessary arguments * mean label smoothing loss * small refactor * fix test Co-authored-by: patrickvonplaten <patrick.v.platen@gmail.com>	2020-11-19 19:18:07 +01:00
				`@@ -1 +0,0 @@`
				`$PYTHON setup.py install # Python command to install the script.`