RoBERTa TensorFlow Tests

Correct slow test
Style
2020-02-04 18:05:22 -05:00 · 2020-02-04 18:05:22 -05:00 · 2020-02-04 18:05:22 -05:00 · 2020-02-04 18:05:22 -05:00 · 2020-02-04 18:05:22 -05:00 · 2020-02-04 17:12:42 -05:00
44 changed files with 1876 additions and 71 deletions
@@ -3,7 +3,7 @@ cd docs
 function deploy_doc(){
 	echo "Creating doc at commit $1 and pushing to folder $2"
 	git checkout $1
-	if [ ! -z "$2" ] 
+	if [ ! -z "$2" ]
 	then
 		if [ -d "$dir/$2" ]; then
 			echo "Directory" $2 "already exists"
@@ -17,7 +17,7 @@ function deploy_doc(){
 	fi
 }

-deploy_doc "master" 
+deploy_doc "master"
 deploy_doc "b33a385" v1.0.0
 deploy_doc "fe02e45" v1.1.0
 deploy_doc "89fd345" v1.2.0
@@ -25,3 +25,4 @@ deploy_doc "fc9faa8" v2.0.0
 deploy_doc "3ddce1d" v2.1.1
 deploy_doc "3616209" v2.2.0
 deploy_doc "d0f8b9a" v2.3.0
+deploy_doc "6664ea9" v2.4.0
@@ -39,12 +39,14 @@ Steps to reproduce the behavior:

 <!-- A clear and concise description of what you would expect to happen. -->

-## Environment
-
-* OS:
-* Python version:
-* PyTorch version:
-* `transformers` version (or branch):
-* Using GPU ?
-* Distributed or parallel setup ?
-* Any other relevant information:
+## Environment info
+<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
+     Don't forget to fill out the missing fields in that output! -->
+     
+- `transformers` version:
+- Platform:
+- Python version:
+- PyTorch version (GPU?):
+- Tensorflow version (GPU?):
+- Using GPU in script?:
+- Using distributed or parallel set-up in script?:
@@ -33,16 +33,21 @@ The tasks I am working on is:
    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
    -->

-## Environment
+## Environment info
+<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
+     Don't forget to fill out the missing fields in that output! -->
+ 
+- `transformers` version:
+- Platform:
+- Python version:
+- PyTorch version (GPU?):
+- Tensorflow version (GPU?):
+- Using GPU in script?:
+- Using distributed or parallel set-up in script?:

-* OS:
-* Python version:
-* PyTorch version:
+<!-- IMPORTANT: which version of the former library do you use? -->
 * `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):
-* `transformers` version (or branch):
-* Using GPU?
-* Distributed or parallel setup?
-* Any other relevant information:
+

 ## Checklist

@@ -41,14 +41,10 @@ Did not find it? :( So we can act quickly on it, please follow these steps:
  less than 30s;
 * Provide the *full* traceback if an exception is raised.

-To get the OS and software versions, execute the following code and copy-paste
-the output:
+To get the OS and software versions automatically, you can run the following command:

-```
-import platform; print("Platform", platform.platform())
-import sys; print("Python", sys.version)
-import torch; print("PyTorch", torch.__version__)
-import tensorflow; print("Tensorflow", tensorflow.__version__)
+```bash
+python transformers-cli env
 ```

 ### Do you want to implement a new model?
@@ -60,7 +60,7 @@ Choose the right framework for every part of a model's lifetime
 | [Quick tour: Share your models ](#Quick-tour-of-model-sharing) | Upload and share your fine-tuned models with the community |
 | [Migrating from pytorch-transformers to transformers](#Migrating-from-pytorch-transformers-to-transformers) | Migrating your code from pytorch-transformers to transformers |
 | [Migrating from pytorch-pretrained-bert to pytorch-transformers](#Migrating-from-pytorch-pretrained-bert-to-transformers) | Migrating your code from pytorch-pretrained-bert to transformers |
-| [Documentation][(v2.3.0)](https://huggingface.co/transformers/v2.3.0)[(v2.2.0/v2.2.1/v2.2.2)](https://huggingface.co/transformers/v2.2.0) [(v2.1.1)](https://huggingface.co/transformers/v2.1.1) [(v2.0.0)](https://huggingface.co/transformers/v2.0.0) [(v1.2.0)](https://huggingface.co/transformers/v1.2.0) [(v1.1.0)](https://huggingface.co/transformers/v1.1.0) [(v1.0.0)](https://huggingface.co/transformers/v1.0.0) [(master)](https://huggingface.co/transformers) | Full API documentation and more |
+| [Documentation][(v2.4.0)](https://huggingface.co/transformers/v2.4.0)[(v2.3.0)](https://huggingface.co/transformers/v2.3.0)[(v2.2.0/v2.2.1/v2.2.2)](https://huggingface.co/transformers/v2.2.0) [(v2.1.1)](https://huggingface.co/transformers/v2.1.1) [(v2.0.0)](https://huggingface.co/transformers/v2.0.0) [(v1.2.0)](https://huggingface.co/transformers/v1.2.0) [(v1.1.0)](https://huggingface.co/transformers/v1.1.0) [(v1.0.0)](https://huggingface.co/transformers/v1.0.0) [(master)](https://huggingface.co/transformers) | Full API documentation and more |

 ## Installation

@@ -26,7 +26,7 @@ author = u'huggingface'
 # The short X.Y version
 version = u''
 # The full version, including alpha/beta/rc tags
-release = u'2.4.0'
+release = u'2.4.1'


 # -- General configuration ---------------------------------------------------
@@ -299,8 +299,8 @@ model = Model2Model.from_pretrained('fine-tuned-weights')
 model.eval()

 # If you have a GPU, put everything on cuda
-question_tensor = encoded_question.to('cuda')
-answer_tensor = encoded_answer.to('cuda')
+question_tensor = question_tensor.to('cuda')
+answer_tensor = answer_tensor.to('cuda')
 model.to('cuda')

 # Predict all tokens
@@ -221,6 +221,7 @@ def main():
        top_k=args.k,
        top_p=args.p,
        repetition_penalty=args.repetition_penalty,
+        do_sample=True,
    )

    # Batch size == 1. to add more examples please use num_return_sequences > 1
@@ -195,6 +195,7 @@ def _rotate_checkpoints(args, checkpoint_prefix="checkpoint", use_mtime=False) -

 def mask_tokens(inputs: torch.Tensor, tokenizer: PreTrainedTokenizer, args) -> Tuple[torch.Tensor, torch.Tensor]:
    """ Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. """
+    inputs = inputs.clone().type(dtype=torch.long)
    labels = inputs.clone()
    # We sample a few tokens in each sequence for masked-LM training (with probability args.mlm_probability defaults to 0.15 in Bert/RoBERTa)
    probability_matrix = torch.full(labels.shape, args.mlm_probability)
@@ -219,6 +219,11 @@ def train(args, train_dataset, model, tokenizer):
                inputs.update({"cls_index": batch[5], "p_mask": batch[6]})
                if args.version_2_with_negative:
                    inputs.update({"is_impossible": batch[7]})
+                if hasattr(model, "config") and hasattr(model.config, "lang2id"):
+                    inputs.update(
+                        {"langs": (torch.ones(batch[0].shape, dtype=torch.int64) * args.lang_id).to(args.device)}
+                    )
+
            outputs = model(**inputs)
            # model outputs are always tuple in transformers (see doc)
            loss = outputs[0]
@@ -330,6 +335,11 @@ def evaluate(args, model, tokenizer, prefix=""):
            # XLNet and XLM use more arguments for their predictions
            if args.model_type in ["xlnet", "xlm"]:
                inputs.update({"cls_index": batch[4], "p_mask": batch[5]})
+                # for lang_id-sensitive xlm models
+                if hasattr(model, "config") and hasattr(model.config, "lang2id"):
+                    inputs.update(
+                        {"langs": (torch.ones(batch[0].shape, dtype=torch.int64) * args.lang_id).to(args.device)}
+                    )

            outputs = model(**inputs)

@@ -635,6 +645,12 @@ def main():
        help="If true, all of the warnings related to data processing will be printed. "
        "A number of warnings are expected for a normal SQuAD evaluation.",
    )
+    parser.add_argument(
+        "--lang_id",
+        default=0,
+        type=int,
+        help="language id of input for language-specific xlm models (see tokenization_xlm.PRETRAINED_INIT_CONFIGURATION)",
+    )

    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
@@ -73,7 +73,7 @@ def read_examples_from_file(data_dir, mode):
                    # Examples could have no label for mode = "test"
                    labels.append("O")
        if words:
-            examples.append(InputExample(guid="%s-%d".format(mode, guid_index), words=words, labels=labels))
+            examples.append(InputExample(guid="{}-{}".format(mode, guid_index), words=words, labels=labels))
    return examples


@@ -0,0 +1,117 @@
+# Swedish BERT Models
+
+The National Library of Sweden / KBLab releases three pretrained language models based on BERT and ALBERT. The models are trained on aproximately 15-20GB of text (200M sentences, 3000M tokens) from various sources (books, news, government publications, swedish wikipedia and internet forums) aiming to provide a representative BERT model for Swedish text. A more complete description will be published later on.
+
+The following three models are currently available:
+
+- **bert-base-swedish-cased** (*v1*) - A BERT trained with the same hyperparameters as first published by Google.
+- **bert-base-swedish-cased-ner** (*experimental*) - a BERT fine-tuned for NER using SUC 3.0.
+- **albert-base-swedish-cased-alpha** (*alpha*) - A first attempt at an ALBERT for Swedish.
+
+All models are cased and trained with whole word masking.
+
+## Files
+
+| **name**                        | **files** |
+|---------------------------------|-----------|
+| bert-base-swedish-cased         | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/vocab.txt), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/pytorch_model.bin) |
+| bert-base-swedish-cased-ner     | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/vocab.txt) [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/pytorch_model.bin) |
+| albert-base-swedish-cased-alpha | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/config.json), [sentencepiece model](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/spiece.model), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/pytorch_model.bin) |
+
+TensorFlow model weights will be released soon.
+
+## Usage requirements / installation instructions
+
+The examples below require Huggingface Transformers 2.4.1 and Pytorch 1.3.1 or greater. For Transformers<2.4.0 the tokenizer must be instantiated manually and the `do_lower_case` flag parameter set to `False` and `keep_accents` to `True` (for ALBERT).
+
+To create an environment where the examples can be run, run the following in an terminal on your OS of choice.
+
+```
+# git clone https://github.com/Kungbib/swedish-bert-models
+# cd swedish-bert-models
+# python3 -m venv venv
+# source venv/bin/activate
+# pip install --upgrade pip
+# pip install -r requirements.txt
+```
+
+### BERT Base Swedish
+
+A standard BERT base for Swedish trained on a variety of sources. Vocabulary size is ~50k. Using Huggingface Transformers the model can be loaded in Python as follows:
+
+```python
+from transformers import AutoModel,AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained('KB/bert-base-swedish-cased')
+model = AutoModel.from_pretrained('KB/bert-base-swedish-cased')
+```
+
+
+### BERT base fine-tuned for Swedish NER
+
+This model is fine-tuned on the SUC 3.0 dataset. Using the Huggingface pipeline the model can be easily instantiated. For Transformer<2.4.1 it seems the tokenizer must be loaded separately to disable lower-casing of input strings:
+
+```python
+from transformers import pipeline
+
+nlp = pipeline('ner', model='KB/bert-base-swedish-cased-ner', tokenizer='KB/bert-base-swedish-cased-ner')
+
+nlp('Idag släpper KB tre språkmodeller.')
+```
+
+Running the Python code above should produce in something like the result below. Entity types used are `TME` for time, `PRS` for personal names, `LOC` for locations, `EVN` for events and `ORG` for organisations. These labels are subject to change.
+
+```python
+[ { 'word': 'Idag', 'score': 0.9998126029968262, 'entity': 'TME' },
+  { 'word': 'KB',   'score': 0.9814832210540771, 'entity': 'ORG' } ]
+```
+
+The BERT tokenizer often splits words into multiple tokens, with the subparts starting with `##`, for example the string `Engelbert kör Volvo till Herrängens fotbollsklubb` gets tokenized as `Engel ##bert kör Volvo till Herr ##ängens fotbolls ##klubb`. To glue parts back together one can use something like this:
+
+```python
+text = 'Engelbert tar Volvon till Tele2 Arena för att titta på Djurgården IF ' +\
+       'som spelar fotboll i VM klockan två på kvällen.'
+
+l = []
+for token in nlp(text):
+    if token['word'].startswith('##'):
+        l[-1]['word'] += token['word'][2:]
+    else:
+        l += [ token ]
+
+print(l)
+```
+
+Which should result in the following (though less cleanly formated):
+
+```python
+[ { 'word': 'Engelbert',     'score': 0.99..., 'entity': 'PRS'},
+  { 'word': 'Volvon',        'score': 0.99..., 'entity': 'OBJ'},
+  { 'word': 'Tele2',         'score': 0.99..., 'entity': 'LOC'},
+  { 'word': 'Arena',         'score': 0.99..., 'entity': 'LOC'},
+  { 'word': 'Djurgården',    'score': 0.99..., 'entity': 'ORG'},
+  { 'word': 'IF',            'score': 0.99..., 'entity': 'ORG'},
+  { 'word': 'VM',            'score': 0.99..., 'entity': 'EVN'},
+  { 'word': 'klockan',       'score': 0.99..., 'entity': 'TME'},
+  { 'word': 'två',           'score': 0.99..., 'entity': 'TME'},
+  { 'word': 'på',            'score': 0.99..., 'entity': 'TME'},
+  { 'word': 'kvällen',       'score': 0.54..., 'entity': 'TME'} ]
+```
+
+### ALBERT base
+
+The easisest way to do this is, again, using Huggingface Transformers:
+
+```python
+from transformers import AutoModel,AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained('KB/albert-base-swedish-cased-alpha'),
+model = AutoModel.from_pretrained('KB/albert-base-swedish-cased-alpha')
+```
+
+## Acknowledgements ❤️
+
+- Resources from Stockholms University, Umeå University and Swedish Language Bank at Gothenburg University were used when fine-tuning BERT for NER.
+- Model pretraining was made partly in-house at the KBLab and partly (for material without active copyright) with the support of Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
+- Models are hosted on S3 by Huggingface 🤗
+
@@ -0,0 +1,117 @@
+# Swedish BERT Models
+
+The National Library of Sweden / KBLab releases three pretrained language models based on BERT and ALBERT. The models are trained on aproximately 15-20GB of text (200M sentences, 3000M tokens) from various sources (books, news, government publications, swedish wikipedia and internet forums) aiming to provide a representative BERT model for Swedish text. A more complete description will be published later on.
+
+The following three models are currently available:
+
+- **bert-base-swedish-cased** (*v1*) - A BERT trained with the same hyperparameters as first published by Google.
+- **bert-base-swedish-cased-ner** (*experimental*) - a BERT fine-tuned for NER using SUC 3.0.
+- **albert-base-swedish-cased-alpha** (*alpha*) - A first attempt at an ALBERT for Swedish.
+
+All models are cased and trained with whole word masking.
+
+## Files
+
+| **name**                        | **files** |
+|---------------------------------|-----------|
+| bert-base-swedish-cased         | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/vocab.txt), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/pytorch_model.bin) |
+| bert-base-swedish-cased-ner     | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/vocab.txt) [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/pytorch_model.bin) |
+| albert-base-swedish-cased-alpha | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/config.json), [sentencepiece model](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/spiece.model), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/pytorch_model.bin) |
+
+TensorFlow model weights will be released soon.
+
+## Usage requirements / installation instructions
+
+The examples below require Huggingface Transformers 2.4.1 and Pytorch 1.3.1 or greater. For Transformers<2.4.0 the tokenizer must be instantiated manually and the `do_lower_case` flag parameter set to `False` and `keep_accents` to `True` (for ALBERT).
+
+To create an environment where the examples can be run, run the following in an terminal on your OS of choice.
+
+```
+# git clone https://github.com/Kungbib/swedish-bert-models
+# cd swedish-bert-models
+# python3 -m venv venv
+# source venv/bin/activate
+# pip install --upgrade pip
+# pip install -r requirements.txt
+```
+
+### BERT Base Swedish
+
+A standard BERT base for Swedish trained on a variety of sources. Vocabulary size is ~50k. Using Huggingface Transformers the model can be loaded in Python as follows:
+
+```python
+from transformers import AutoModel,AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained('KB/bert-base-swedish-cased')
+model = AutoModel.from_pretrained('KB/bert-base-swedish-cased')
+```
+
+
+### BERT base fine-tuned for Swedish NER
+
+This model is fine-tuned on the SUC 3.0 dataset. Using the Huggingface pipeline the model can be easily instantiated. For Transformer<2.4.1 it seems the tokenizer must be loaded separately to disable lower-casing of input strings:
+
+```python
+from transformers import pipeline
+
+nlp = pipeline('ner', model='KB/bert-base-swedish-cased-ner', tokenizer='KB/bert-base-swedish-cased-ner')
+
+nlp('Idag släpper KB tre språkmodeller.')
+```
+
+Running the Python code above should produce in something like the result below. Entity types used are `TME` for time, `PRS` for personal names, `LOC` for locations, `EVN` for events and `ORG` for organisations. These labels are subject to change.
+
+```python
+[ { 'word': 'Idag', 'score': 0.9998126029968262, 'entity': 'TME' },
+  { 'word': 'KB',   'score': 0.9814832210540771, 'entity': 'ORG' } ]
+```
+
+The BERT tokenizer often splits words into multiple tokens, with the subparts starting with `##`, for example the string `Engelbert kör Volvo till Herrängens fotbollsklubb` gets tokenized as `Engel ##bert kör Volvo till Herr ##ängens fotbolls ##klubb`. To glue parts back together one can use something like this:
+
+```python
+text = 'Engelbert tar Volvon till Tele2 Arena för att titta på Djurgården IF ' +\
+       'som spelar fotboll i VM klockan två på kvällen.'
+
+l = []
+for token in nlp(text):
+    if token['word'].startswith('##'):
+        l[-1]['word'] += token['word'][2:]
+    else:
+        l += [ token ]
+
+print(l)
+```
+
+Which should result in the following (though less cleanly formated):
+
+```python
+[ { 'word': 'Engelbert',     'score': 0.99..., 'entity': 'PRS'},
+  { 'word': 'Volvon',        'score': 0.99..., 'entity': 'OBJ'},
+  { 'word': 'Tele2',         'score': 0.99..., 'entity': 'LOC'},
+  { 'word': 'Arena',         'score': 0.99..., 'entity': 'LOC'},
+  { 'word': 'Djurgården',    'score': 0.99..., 'entity': 'ORG'},
+  { 'word': 'IF',            'score': 0.99..., 'entity': 'ORG'},
+  { 'word': 'VM',            'score': 0.99..., 'entity': 'EVN'},
+  { 'word': 'klockan',       'score': 0.99..., 'entity': 'TME'},
+  { 'word': 'två',           'score': 0.99..., 'entity': 'TME'},
+  { 'word': 'på',            'score': 0.99..., 'entity': 'TME'},
+  { 'word': 'kvällen',       'score': 0.54..., 'entity': 'TME'} ]
+```
+
+### ALBERT base
+
+The easisest way to do this is, again, using Huggingface Transformers:
+
+```python
+from transformers import AutoModel,AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained('KB/albert-base-swedish-cased-alpha'),
+model = AutoModel.from_pretrained('KB/albert-base-swedish-cased-alpha')
+```
+
+## Acknowledgements ❤️
+
+- Resources from Stockholms University, Umeå University and Swedish Language Bank at Gothenburg University were used when fine-tuning BERT for NER.
+- Model pretraining was made partly in-house at the KBLab and partly (for material without active copyright) with the support of Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
+- Models are hosted on S3 by Huggingface 🤗
+
@@ -0,0 +1,117 @@
+# Swedish BERT Models
+
+The National Library of Sweden / KBLab releases three pretrained language models based on BERT and ALBERT. The models are trained on aproximately 15-20GB of text (200M sentences, 3000M tokens) from various sources (books, news, government publications, swedish wikipedia and internet forums) aiming to provide a representative BERT model for Swedish text. A more complete description will be published later on.
+
+The following three models are currently available:
+
+- **bert-base-swedish-cased** (*v1*) - A BERT trained with the same hyperparameters as first published by Google.
+- **bert-base-swedish-cased-ner** (*experimental*) - a BERT fine-tuned for NER using SUC 3.0.
+- **albert-base-swedish-cased-alpha** (*alpha*) - A first attempt at an ALBERT for Swedish.
+
+All models are cased and trained with whole word masking.
+
+## Files
+
+| **name**                        | **files** |
+|---------------------------------|-----------|
+| bert-base-swedish-cased         | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/vocab.txt), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/pytorch_model.bin) |
+| bert-base-swedish-cased-ner     | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/vocab.txt) [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/pytorch_model.bin) |
+| albert-base-swedish-cased-alpha | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/config.json), [sentencepiece model](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/spiece.model), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/pytorch_model.bin) |
+
+TensorFlow model weights will be released soon.
+
+## Usage requirements / installation instructions
+
+The examples below require Huggingface Transformers 2.4.1 and Pytorch 1.3.1 or greater. For Transformers<2.4.0 the tokenizer must be instantiated manually and the `do_lower_case` flag parameter set to `False` and `keep_accents` to `True` (for ALBERT).
+
+To create an environment where the examples can be run, run the following in an terminal on your OS of choice.
+
+```
+# git clone https://github.com/Kungbib/swedish-bert-models
+# cd swedish-bert-models
+# python3 -m venv venv
+# source venv/bin/activate
+# pip install --upgrade pip
+# pip install -r requirements.txt
+```
+
+### BERT Base Swedish
+
+A standard BERT base for Swedish trained on a variety of sources. Vocabulary size is ~50k. Using Huggingface Transformers the model can be loaded in Python as follows:
+
+```python
+from transformers import AutoModel,AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained('KB/bert-base-swedish-cased')
+model = AutoModel.from_pretrained('KB/bert-base-swedish-cased')
+```
+
+
+### BERT base fine-tuned for Swedish NER
+
+This model is fine-tuned on the SUC 3.0 dataset. Using the Huggingface pipeline the model can be easily instantiated. For Transformer<2.4.1 it seems the tokenizer must be loaded separately to disable lower-casing of input strings:
+
+```python
+from transformers import pipeline
+
+nlp = pipeline('ner', model='KB/bert-base-swedish-cased-ner', tokenizer='KB/bert-base-swedish-cased-ner')
+
+nlp('Idag släpper KB tre språkmodeller.')
+```
+
+Running the Python code above should produce in something like the result below. Entity types used are `TME` for time, `PRS` for personal names, `LOC` for locations, `EVN` for events and `ORG` for organisations. These labels are subject to change.
+
+```python
+[ { 'word': 'Idag', 'score': 0.9998126029968262, 'entity': 'TME' },
+  { 'word': 'KB',   'score': 0.9814832210540771, 'entity': 'ORG' } ]
+```
+
+The BERT tokenizer often splits words into multiple tokens, with the subparts starting with `##`, for example the string `Engelbert kör Volvo till Herrängens fotbollsklubb` gets tokenized as `Engel ##bert kör Volvo till Herr ##ängens fotbolls ##klubb`. To glue parts back together one can use something like this:
+
+```python
+text = 'Engelbert tar Volvon till Tele2 Arena för att titta på Djurgården IF ' +\
+       'som spelar fotboll i VM klockan två på kvällen.'
+
+l = []
+for token in nlp(text):
+    if token['word'].startswith('##'):
+        l[-1]['word'] += token['word'][2:]
+    else:
+        l += [ token ]
+
+print(l)
+```
+
+Which should result in the following (though less cleanly formated):
+
+```python
+[ { 'word': 'Engelbert',     'score': 0.99..., 'entity': 'PRS'},
+  { 'word': 'Volvon',        'score': 0.99..., 'entity': 'OBJ'},
+  { 'word': 'Tele2',         'score': 0.99..., 'entity': 'LOC'},
+  { 'word': 'Arena',         'score': 0.99..., 'entity': 'LOC'},
+  { 'word': 'Djurgården',    'score': 0.99..., 'entity': 'ORG'},
+  { 'word': 'IF',            'score': 0.99..., 'entity': 'ORG'},
+  { 'word': 'VM',            'score': 0.99..., 'entity': 'EVN'},
+  { 'word': 'klockan',       'score': 0.99..., 'entity': 'TME'},
+  { 'word': 'två',           'score': 0.99..., 'entity': 'TME'},
+  { 'word': 'på',            'score': 0.99..., 'entity': 'TME'},
+  { 'word': 'kvällen',       'score': 0.54..., 'entity': 'TME'} ]
+```
+
+### ALBERT base
+
+The easisest way to do this is, again, using Huggingface Transformers:
+
+```python
+from transformers import AutoModel,AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained('KB/albert-base-swedish-cased-alpha'),
+model = AutoModel.from_pretrained('KB/albert-base-swedish-cased-alpha')
+```
+
+## Acknowledgements ❤️
+
+- Resources from Stockholms University, Umeå University and Swedish Language Bank at Gothenburg University were used when fine-tuning BERT for NER.
+- Model pretraining was made partly in-house at the KBLab and partly (for material without active copyright) with the support of Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
+- Models are hosted on S3 by Huggingface 🤗
+
@@ -0,0 +1,114 @@
+# UmBERTo Commoncrawl Cased
+
+[UmBERTo](https://github.com/musixmatchresearch/umberto) is a Roberta-based Language Model trained on large Italian Corpora and uses two innovative approaches: SentencePiece and Whole Word Masking. Now available at [github.com/huggingface/transformers](https://huggingface.co/Musixmatch/umberto-commoncrawl-cased-v1)
+
+<p align="center">
+    <img src="https://user-images.githubusercontent.com/7140210/72913702-d55a8480-3d3d-11ea-99fc-f2ef29af4e72.jpg" width="700"> </br>
+    Marco Lodola, Monument to Umberto Eco, Alessandria 2019
+</p>
+
+## Dataset
+UmBERTo-Commoncrawl-Cased utilizes the Italian subcorpus of [OSCAR](https://traces1.inria.fr/oscar/) as training set of the language model. We used deduplicated version of the Italian corpus that consists in 70 GB of plain text data, 210M sentences with 11B words where the sentences have been filtered and shuffled at line level in order to be used for NLP research.
+
+## Pre-trained model
+
+| Model | WWM | Cased | Tokenizer | Vocab Size  | Train Steps |  Download |
+| ------ | ------ | ------ | ------ | ------ |------ | ------ |
+| `umberto-commoncrawl-cased-v1` | YES | YES | SPM | 32K | 125k | [Link](http://bit.ly/35zO7GH) |
+
+This model was trained with [SentencePiece](https://github.com/google/sentencepiece) and Whole Word Masking.
+
+## Downstream Tasks
+These results refers to umberto-commoncrawl-cased model. All details are at [Umberto](https://github.com/musixmatchresearch/umberto) Official Page.
+
+#### Named Entity Recognition (NER)
+
+| Dataset | F1 | Precision | Recall | Accuracy |
+| ------ | ------ | ------ |  ------ |  ------ |
+| **ICAB-EvalITA07** | **87.565**  | 86.596  | 88.556  | 98.690 | 
+| **WikiNER-ITA** | **92.531**  | 92.509 | 92.553 | 99.136 | 
+
+#### Part of Speech (POS)
+
+| Dataset | F1 | Precision | Recall | Accuracy |
+| ------ | ------ | ------ |  ------ |  ------ |
+| **UD_Italian-ISDT** | 98.870  | 98.861 | 98.879 | **98.977** | 
+| **UD_Italian-ParTUT** | 98.786 | 98.812 |  98.760 | **98.903** | 
+
+
+
+## Usage
+
+##### Load UmBERTo with AutoModel, Autotokenizer:
+
+```python
+
+import torch
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("Musixmatch/umberto-commoncrawl-cased-v1")
+umberto = AutoModel.from_pretrained("Musixmatch/umberto-commoncrawl-cased-v1")
+
+encoded_input = tokenizer.encode("Umberto Eco è stato un grande scrittore")
+input_ids = torch.tensor(encoded_input).unsqueeze(0)  # Batch size 1
+outputs = umberto(input_ids)
+last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output
+```
+
+##### Predict masked token:
+
+```python
+from transformers import pipeline
+
+fill_mask = pipeline(
+	"fill-mask",
+	model="Musixmatch/umberto-commoncrawl-cased-v1",
+	tokenizer="Musixmatch/umberto-commoncrawl-cased-v1"
+)
+
+result = fill_mask("Umberto Eco è <mask> un grande scrittore")
+# {'sequence': '<s> Umberto Eco è considerato un grande scrittore</s>', 'score': 0.18599839508533478, 'token': 5032}
+# {'sequence': '<s> Umberto Eco è stato un grande scrittore</s>', 'score': 0.17816807329654694, 'token': 471}
+# {'sequence': '<s> Umberto Eco è sicuramente un grande scrittore</s>', 'score': 0.16565583646297455, 'token': 2654}
+# {'sequence': '<s> Umberto Eco è indubbiamente un grande scrittore</s>', 'score': 0.0932890921831131, 'token': 17908}
+# {'sequence': '<s> Umberto Eco è certamente un grande scrittore</s>', 'score': 0.054701317101716995, 'token': 5269}
+```
+
+
+## Citation
+All of the original datasets are publicly available or were released with the owners' grant. The datasets are all released under a CC0 or CCBY license.
+
+* UD Italian-ISDT Dataset [Github](https://github.com/UniversalDependencies/UD_Italian-ISDT)
+* UD Italian-ParTUT Dataset [Github](https://github.com/UniversalDependencies/UD_Italian-ParTUT)
+* I-CAB (Italian Content Annotation Bank), EvalITA [Page](http://www.evalita.it/)
+* WIKINER [Page](https://figshare.com/articles/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500) , [Paper](https://www.sciencedirect.com/science/article/pii/S0004370212000276?via%3Dihub)
+
+```
+@inproceedings {magnini2006annotazione,
+	title = {Annotazione di contenuti concettuali in un corpus italiano: I - CAB},
+	author = {Magnini,Bernardo and Cappelli,Amedeo and Pianta,Emanuele and Speranza,Manuela and Bartalesi Lenzi,V and Sprugnoli,Rachele and Romano,Lorenza and Girardi,Christian and Negri,Matteo},
+	booktitle = {Proc.of SILFI 2006},
+	year = {2006}
+}
+@inproceedings {magnini2006cab,
+	title = {I - CAB: the Italian Content Annotation Bank.},
+	author = {Magnini,Bernardo and Pianta,Emanuele and Girardi,Christian and Negri,Matteo and Romano,Lorenza and Speranza,Manuela and Lenzi,Valentina Bartalesi and Sprugnoli,Rachele},
+	booktitle = {LREC},
+	pages = {963--968},
+	year = {2006},
+	organization = {Citeseer}
+}
+```
+
+## Authors
+
+**Loreto Parisi**: `loreto at musixmatch dot com`, [loretoparisi](https://github.com/loretoparisi)<br>
+**Simone Francia**: `simone.francia at musixmatch dot com`, [simonefrancia](https://github.com/simonefrancia)<br>
+**Paolo Magnani**: `paul.magnani95 at gmail dot com`, [paulthemagno](https://github.com/paulthemagno)<br>
+
+## About Musixmatch AI
+![Musxmatch Ai mac app icon-128](https://user-images.githubusercontent.com/163333/72244273-396aa380-35ee-11ea-894b-4ea48230c02b.png)<br>
+We do Machine Learning and Artificial Intelligence @[musixmatch](https://twitter.com/Musixmatch)<br>
+Follow us on [Twitter](https://twitter.com/musixmatchai) [Github](https://github.com/musixmatchresearch)
+
+
@@ -0,0 +1,113 @@
+# UmBERTo Wikipedia Uncased
+
+[UmBERTo](https://github.com/musixmatchresearch/umberto) is a Roberta-based Language Model trained on large Italian Corpora and uses two innovative approaches: SentencePiece and Whole Word Masking. Now available at [github.com/huggingface/transformers](https://huggingface.co/Musixmatch/umberto-commoncrawl-cased-v1)
+
+<p align="center">
+    <img src="https://user-images.githubusercontent.com/7140210/72913702-d55a8480-3d3d-11ea-99fc-f2ef29af4e72.jpg" width="700"> </br>
+    Marco Lodola, Monument to Umberto Eco, Alessandria 2019
+</p>
+
+## Dataset
+UmBERTo-Wikipedia-Uncased Training is trained on a relative small corpus (~7GB) extracted from [Wikipedia-ITA](https://linguatools.org/tools/corpora/wikipedia-monolingual-corpora/).
+
+## Pre-trained model
+
+| Model | WWM | Cased | Tokenizer | Vocab Size  | Train Steps |  Download |
+| ------ | ------ | ------ | ------ | ------ |------ | ------ |
+| `umberto-wikipedia-uncased-v1` | YES | YES | SPM | 32K | 100k | [Link](http://bit.ly/35wbSj6) |
+
+This model was trained with [SentencePiece](https://github.com/google/sentencepiece) and Whole Word Masking.
+
+## Downstream Tasks
+These results refers to umberto-wikipedia-uncased model. All details are at [Umberto](https://github.com/musixmatchresearch/umberto) Official Page.
+
+#### Named Entity Recognition (NER)
+
+| Dataset | F1 | Precision | Recall | Accuracy |
+| ------ | ------ | ------ |  ------ |  ----- |
+| **ICAB-EvalITA07** | **86.240** | 85.939 | 86.544 | 98.534 | 
+| **WikiNER-ITA** | **90.483** | 90.328 | 90.638 | 98.661 | 
+
+#### Part of Speech (POS)
+
+| Dataset | F1 | Precision | Recall | Accuracy |
+| ------ | ------ | ------ |  ------ |  ------ |
+| **UD_Italian-ISDT** | 98.563  | 98.508 | 98.618 | **98.717** | 
+| **UD_Italian-ParTUT** | 97.810 | 97.835 |  97.784 | **98.060** | 
+
+
+
+## Usage
+
+##### Load UmBERTo Wikipedia Uncased with AutoModel, Autotokenizer:
+
+```python
+
+import torch
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("Musixmatch/umberto-wikipedia-uncased-v1")
+umberto = AutoModel.from_pretrained("Musixmatch/umberto-wikipedia-uncased-v1")
+
+encoded_input = tokenizer.encode("Umberto Eco è stato un grande scrittore")
+input_ids = torch.tensor(encoded_input).unsqueeze(0)  # Batch size 1
+outputs = umberto(input_ids)
+last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output
+```
+
+##### Predict masked token:
+
+```python
+from transformers import pipeline
+
+fill_mask = pipeline(
+	"fill-mask",
+	model="Musixmatch/umberto-wikipedia-uncased-v1",
+	tokenizer="Musixmatch/umberto-wikipedia-uncased-v1"
+)
+
+result = fill_mask("Umberto Eco è <mask> un grande scrittore")
+# {'sequence': '<s> umberto eco è stato un grande scrittore</s>', 'score': 0.5784581303596497, 'token': 361}
+# {'sequence': '<s> umberto eco è anche un grande scrittore</s>', 'score': 0.33813193440437317, 'token': 269}
+# {'sequence': '<s> umberto eco è considerato un grande scrittore</s>', 'score': 0.027196012437343597, 'token': 3236}
+# {'sequence': '<s> umberto eco è diventato un grande scrittore</s>', 'score': 0.013716378249228, 'token': 5742}
+# {'sequence': '<s> umberto eco è inoltre un grande scrittore</s>', 'score': 0.010662357322871685, 'token': 1030}
+```
+
+
+## Citation
+All of the original datasets are publicly available or were released with the owners' grant. The datasets are all released under a CC0 or CCBY license.
+
+* UD Italian-ISDT Dataset [Github](https://github.com/UniversalDependencies/UD_Italian-ISDT)
+* UD Italian-ParTUT Dataset [Github](https://github.com/UniversalDependencies/UD_Italian-ParTUT)
+* I-CAB (Italian Content Annotation Bank), EvalITA [Page](http://www.evalita.it/)
+* WIKINER [Page](https://figshare.com/articles/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500) , [Paper](https://www.sciencedirect.com/science/article/pii/S0004370212000276?via%3Dihub)
+
+```
+@inproceedings {magnini2006annotazione,
+	title = {Annotazione di contenuti concettuali in un corpus italiano: I - CAB},
+	author = {Magnini,Bernardo and Cappelli,Amedeo and Pianta,Emanuele and Speranza,Manuela and Bartalesi Lenzi,V and Sprugnoli,Rachele and Romano,Lorenza and Girardi,Christian and Negri,Matteo},
+	booktitle = {Proc.of SILFI 2006},
+	year = {2006}
+}
+@inproceedings {magnini2006cab,
+	title = {I - CAB: the Italian Content Annotation Bank.},
+	author = {Magnini,Bernardo and Pianta,Emanuele and Girardi,Christian and Negri,Matteo and Romano,Lorenza and Speranza,Manuela and Lenzi,Valentina Bartalesi and Sprugnoli,Rachele},
+	booktitle = {LREC},
+	pages = {963--968},
+	year = {2006},
+	organization = {Citeseer}
+}
+```
+
+## Authors
+
+**Loreto Parisi**: `loreto at musixmatch dot com`, [loretoparisi](https://github.com/loretoparisi)<br>
+**Simone Francia**: `simone.francia at musixmatch dot com`, [simonefrancia](https://github.com/simonefrancia)<br>
+**Paolo Magnani**: `paul.magnani95 at gmail dot com`, [paulthemagno](https://github.com/paulthemagno)<br>
+
+## About Musixmatch AI
+![Musxmatch Ai mac app icon-128](https://user-images.githubusercontent.com/163333/72244273-396aa380-35ee-11ea-894b-4ea48230c02b.png)<br>
+We do Machine Learning and Artificial Intelligence @[musixmatch](https://twitter.com/Musixmatch)<br>
+Follow us on [Twitter](https://twitter.com/musixmatchai) [Github](https://github.com/musixmatchresearch)
+
@@ -0,0 +1,66 @@
+# 🤗 + 📚 dbmdz German BERT models
+
+In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
+Library open sources another German BERT models 🎉
+
+# German BERT
+
+## Stats
+
+In addition to the recently released [German BERT](https://deepset.ai/german-bert)
+model by [deepset](https://deepset.ai/) we provide another German-language model.
+
+The source data for the model consists of a recent Wikipedia dump, EU Bookshop corpus,
+Open Subtitles, CommonCrawl, ParaCrawl and News Crawl. This results in a dataset with
+a size of 16GB and 2,350,234,427 tokens.
+
+For sentence splitting, we use [spacy](https://spacy.io/). Our preprocessing steps
+(sentence piece model for vocab generation) follow those used for training
+[SciBERT](https://github.com/allenai/scibert). The model is trained with an initial
+sequence length of 512 subwords and was performed for 1.5M steps.
+
+This release includes both cased and uncased models.
+
+## Model weights
+
+Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
+compatible weights are available. If you need access to TensorFlow checkpoints,
+please raise an issue!
+
+| Model                            | Downloads
+| -------------------------------- | ---------------------------------------------------------------------------------------------------------------
+| `bert-base-german-dbmdz-cased`   | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-config.json) • [`pytorch_model.bin`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-pytorch_model.bin) • [`vocab.txt`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-vocab.txt)
+| `bert-base-german-dbmdz-uncased` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-config.json) • [`pytorch_model.bin`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-pytorch_model.bin) • [`vocab.txt`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-vocab.txt)
+
+## Usage
+
+With Transformers >= 2.3 our German BERT models can be loaded like:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased")
+model = AutoModel.from_pretrained("dbmdz/bert-base-german-cased")
+```
+
+## Results
+
+For results on downstream tasks like NER or PoS tagging, please refer to
+[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
+
+# Huggingface model hub
+
+All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
+
+# Contact (Bugs, Feedback, Contribution and more)
+
+For questions about our BERT models just open an issue
+[here](https://github.com/dbmdz/berts/issues/new) 🤗
+
+# Acknowledgments
+
+Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
+Thanks for providing access to the TFRC ❤️
+
+Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
+it is possible to download both cased and uncased models from their S3 storage 🤗
@@ -0,0 +1,66 @@
+# 🤗 + 📚 dbmdz German BERT models
+
+In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
+Library open sources another German BERT models 🎉
+
+# German BERT
+
+## Stats
+
+In addition to the recently released [German BERT](https://deepset.ai/german-bert)
+model by [deepset](https://deepset.ai/) we provide another German-language model.
+
+The source data for the model consists of a recent Wikipedia dump, EU Bookshop corpus,
+Open Subtitles, CommonCrawl, ParaCrawl and News Crawl. This results in a dataset with
+a size of 16GB and 2,350,234,427 tokens.
+
+For sentence splitting, we use [spacy](https://spacy.io/). Our preprocessing steps
+(sentence piece model for vocab generation) follow those used for training
+[SciBERT](https://github.com/allenai/scibert). The model is trained with an initial
+sequence length of 512 subwords and was performed for 1.5M steps.
+
+This release includes both cased and uncased models.
+
+## Model weights
+
+Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
+compatible weights are available. If you need access to TensorFlow checkpoints,
+please raise an issue!
+
+| Model                            | Downloads
+| -------------------------------- | ---------------------------------------------------------------------------------------------------------------
+| `bert-base-german-dbmdz-cased`   | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-config.json) • [`pytorch_model.bin`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-pytorch_model.bin) • [`vocab.txt`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-vocab.txt)
+| `bert-base-german-dbmdz-uncased` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-config.json) • [`pytorch_model.bin`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-pytorch_model.bin) • [`vocab.txt`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-vocab.txt)
+
+## Usage
+
+With Transformers >= 2.3 our German BERT models can be loaded like:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased")
+model = AutoModel.from_pretrained("dbmdz/bert-base-german-cased")
+```
+
+## Results
+
+For results on downstream tasks like NER or PoS tagging, please refer to
+[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
+
+# Huggingface model hub
+
+All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
+
+# Contact (Bugs, Feedback, Contribution and more)
+
+For questions about our BERT models just open an issue
+[here](https://github.com/dbmdz/berts/issues/new) 🤗
+
+# Acknowledgments
+
+Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
+Thanks for providing access to the TFRC ❤️
+
+Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
+it is possible to download both cased and uncased models from their S3 storage 🤗
@@ -0,0 +1,73 @@
+# 🤗 + 📚 dbmdz BERT models
+
+In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
+Library open sources Italian BERT models 🎉
+
+# Italian BERT
+
+The source data for the Italian BERT model consists of a recent Wikipedia dump and
+various texts from the [OPUS corpora](http://opus.nlpl.eu/) collection. The final
+training corpus has a size of 13GB and 2,050,057,573 tokens.
+
+For sentence splitting, we use NLTK (faster compared to spacy).
+Our cased and uncased models are training with an initial sequence length of 512
+subwords for ~2-3M steps.
+
+For the XXL Italian models, we use the same training data from OPUS and extend
+it with data from the Italian part of the [OSCAR corpus](https://traces1.inria.fr/oscar/).
+Thus, the final training corpus has a size of 81GB and 13,138,379,147 tokens.
+
+## Model weights
+
+Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
+compatible weights are available. If you need access to TensorFlow checkpoints,
+please raise an issue!
+
+| Model                                   | Downloads
+| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------
+| `dbmdz/bert-base-italian-cased`         | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/config.json)       • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/pytorch_model.bin)       • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/vocab.txt)
+| `dbmdz/bert-base-italian-uncased`       | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/config.json)     • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/pytorch_model.bin)     • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/vocab.txt)
+| `dbmdz/bert-base-italian-xxl-cased`     | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/config.json)   • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/pytorch_model.bin)   • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/vocab.txt)
+| `dbmdz/bert-base-italian-xxl-uncased`   | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/vocab.txt)
+
+## Results
+
+For results on downstream tasks like NER or PoS tagging, please refer to
+[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
+
+## Usage
+
+With Transformers >= 2.3 our Italian BERT models can be loaded like:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-cased")
+model = AutoModel.from_pretrained("dbmdz/bert-base-italian-cased")
+```
+
+To load the (recommended) Italian XXL BERT models, just use:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
+model = AutoModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
+```
+
+# Huggingface model hub
+
+All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
+
+# Contact (Bugs, Feedback, Contribution and more)
+
+For questions about our BERT models just open an issue
+[here](https://github.com/dbmdz/berts/issues/new) 🤗
+
+# Acknowledgments
+
+Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
+Thanks for providing access to the TFRC ❤️
+
+Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
+it is possible to download both cased and uncased models from their S3 storage 🤗
@@ -0,0 +1,73 @@
+# 🤗 + 📚 dbmdz BERT models
+
+In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
+Library open sources Italian BERT models 🎉
+
+# Italian BERT
+
+The source data for the Italian BERT model consists of a recent Wikipedia dump and
+various texts from the [OPUS corpora](http://opus.nlpl.eu/) collection. The final
+training corpus has a size of 13GB and 2,050,057,573 tokens.
+
+For sentence splitting, we use NLTK (faster compared to spacy).
+Our cased and uncased models are training with an initial sequence length of 512
+subwords for ~2-3M steps.
+
+For the XXL Italian models, we use the same training data from OPUS and extend
+it with data from the Italian part of the [OSCAR corpus](https://traces1.inria.fr/oscar/).
+Thus, the final training corpus has a size of 81GB and 13,138,379,147 tokens.
+
+## Model weights
+
+Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
+compatible weights are available. If you need access to TensorFlow checkpoints,
+please raise an issue!
+
+| Model                                   | Downloads
+| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------
+| `dbmdz/bert-base-italian-cased`         | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/config.json)       • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/pytorch_model.bin)       • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/vocab.txt)
+| `dbmdz/bert-base-italian-uncased`       | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/config.json)     • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/pytorch_model.bin)     • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/vocab.txt)
+| `dbmdz/bert-base-italian-xxl-cased`     | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/config.json)   • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/pytorch_model.bin)   • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/vocab.txt)
+| `dbmdz/bert-base-italian-xxl-uncased`   | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/vocab.txt)
+
+## Results
+
+For results on downstream tasks like NER or PoS tagging, please refer to
+[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
+
+## Usage
+
+With Transformers >= 2.3 our Italian BERT models can be loaded like:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-cased")
+model = AutoModel.from_pretrained("dbmdz/bert-base-italian-cased")
+```
+
+To load the (recommended) Italian XXL BERT models, just use:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
+model = AutoModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
+```
+
+# Huggingface model hub
+
+All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
+
+# Contact (Bugs, Feedback, Contribution and more)
+
+For questions about our BERT models just open an issue
+[here](https://github.com/dbmdz/berts/issues/new) 🤗
+
+# Acknowledgments
+
+Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
+Thanks for providing access to the TFRC ❤️
+
+Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
+it is possible to download both cased and uncased models from their S3 storage 🤗
@@ -0,0 +1,73 @@
+# 🤗 + 📚 dbmdz BERT models
+
+In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
+Library open sources Italian BERT models 🎉
+
+# Italian BERT
+
+The source data for the Italian BERT model consists of a recent Wikipedia dump and
+various texts from the [OPUS corpora](http://opus.nlpl.eu/) collection. The final
+training corpus has a size of 13GB and 2,050,057,573 tokens.
+
+For sentence splitting, we use NLTK (faster compared to spacy).
+Our cased and uncased models are training with an initial sequence length of 512
+subwords for ~2-3M steps.
+
+For the XXL Italian models, we use the same training data from OPUS and extend
+it with data from the Italian part of the [OSCAR corpus](https://traces1.inria.fr/oscar/).
+Thus, the final training corpus has a size of 81GB and 13,138,379,147 tokens.
+
+## Model weights
+
+Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
+compatible weights are available. If you need access to TensorFlow checkpoints,
+please raise an issue!
+
+| Model                                   | Downloads
+| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------
+| `dbmdz/bert-base-italian-cased`         | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/config.json)       • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/pytorch_model.bin)       • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/vocab.txt)
+| `dbmdz/bert-base-italian-uncased`       | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/config.json)     • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/pytorch_model.bin)     • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/vocab.txt)
+| `dbmdz/bert-base-italian-xxl-cased`     | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/config.json)   • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/pytorch_model.bin)   • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/vocab.txt)
+| `dbmdz/bert-base-italian-xxl-uncased`   | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/vocab.txt)
+
+## Results
+
+For results on downstream tasks like NER or PoS tagging, please refer to
+[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
+
+## Usage
+
+With Transformers >= 2.3 our Italian BERT models can be loaded like:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-cased")
+model = AutoModel.from_pretrained("dbmdz/bert-base-italian-cased")
+```
+
+To load the (recommended) Italian XXL BERT models, just use:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
+model = AutoModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
+```
+
+# Huggingface model hub
+
+All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
+
+# Contact (Bugs, Feedback, Contribution and more)
+
+For questions about our BERT models just open an issue
+[here](https://github.com/dbmdz/berts/issues/new) 🤗
+
+# Acknowledgments
+
+Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
+Thanks for providing access to the TFRC ❤️
+
+Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
+it is possible to download both cased and uncased models from their S3 storage 🤗
@@ -0,0 +1,73 @@
+# 🤗 + 📚 dbmdz BERT models
+
+In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
+Library open sources Italian BERT models 🎉
+
+# Italian BERT
+
+The source data for the Italian BERT model consists of a recent Wikipedia dump and
+various texts from the [OPUS corpora](http://opus.nlpl.eu/) collection. The final
+training corpus has a size of 13GB and 2,050,057,573 tokens.
+
+For sentence splitting, we use NLTK (faster compared to spacy).
+Our cased and uncased models are training with an initial sequence length of 512
+subwords for ~2-3M steps.
+
+For the XXL Italian models, we use the same training data from OPUS and extend
+it with data from the Italian part of the [OSCAR corpus](https://traces1.inria.fr/oscar/).
+Thus, the final training corpus has a size of 81GB and 13,138,379,147 tokens.
+
+## Model weights
+
+Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
+compatible weights are available. If you need access to TensorFlow checkpoints,
+please raise an issue!
+
+| Model                                   | Downloads
+| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------
+| `dbmdz/bert-base-italian-cased`         | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/config.json)       • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/pytorch_model.bin)       • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/vocab.txt)
+| `dbmdz/bert-base-italian-uncased`       | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/config.json)     • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/pytorch_model.bin)     • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/vocab.txt)
+| `dbmdz/bert-base-italian-xxl-cased`     | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/config.json)   • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/pytorch_model.bin)   • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/vocab.txt)
+| `dbmdz/bert-base-italian-xxl-uncased`   | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/vocab.txt)
+
+## Results
+
+For results on downstream tasks like NER or PoS tagging, please refer to
+[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
+
+## Usage
+
+With Transformers >= 2.3 our Italian BERT models can be loaded like:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-cased")
+model = AutoModel.from_pretrained("dbmdz/bert-base-italian-cased")
+```
+
+To load the (recommended) Italian XXL BERT models, just use:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
+model = AutoModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
+```
+
+# Huggingface model hub
+
+All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
+
+# Contact (Bugs, Feedback, Contribution and more)
+
+For questions about our BERT models just open an issue
+[here](https://github.com/dbmdz/berts/issues/new) 🤗
+
+# Acknowledgments
+
+Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
+Thanks for providing access to the TFRC ❤️
+
+Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
+it is possible to download both cased and uncased models from their S3 storage 🤗
@@ -0,0 +1,46 @@
+# Multilingual + Dutch SQuAD2.0
+
+This model is the multilingual model provided by the Google research team with a fine-tuned dutch Q&A downstream task.
+
+## Details of the language model(bert-base-multilingual-cased)
+
+Language model ([**bert-base-multilingual-cased**](https://github.com/google-research/bert/blob/master/multilingual.md)):
+12-layer, 768-hidden, 12-heads, 110M parameters.
+Trained on cased text in the top 104 languages with the largest Wikipedias.
+
+## Details of the downstream task - Dataset
+Using the `mtranslate` Python module, [**SQuAD2.0**](https://rajpurkar.github.io/SQuAD-explorer/) was machine-translated. In order to find the start tokens the direct translations of the answers were searched in the corresponding paragraphs. Since the answer could not always be found in the text, due to the different translations depending on the context (missing context in the pure answer), a loss of question-answer examples occurred. This is a potential problem where errors can occur in the data set (but in the end it was a quick and dirty solution that worked well enough for my task).
+
+| Dataset                | # Q&A |
+| ---------------------- | ----- |
+| SQuAD2.0 Train         | 130 K |
+| Dutch SQuAD2.0 Train   | 99  K |
+| SQuAD2.0 Dev           | 12  K |
+| Dutch SQuAD2.0 Dev     | 10  K |
+
+## Model training
+
+The model was trained on a Tesla V100 GPU with the following command:
+
+```python
+export SQUAD_DIR=path/to/nl_squad
+
+python run_squad.py \
+  --model_type bert \
+  --model_name_or_path bert-base-multilingual-cased \
+  --version_2_with_negative \
+  --do_train \
+  --do_eval \
+  --train_file $SQUAD_DIR/train_nl-v2.0.json \
+  --predict_file $SQUAD_DIR/dev_nl-v2.0.json \
+  --per_gpu_train_batch_size 12 \
+  --learning_rate 3e-5 \
+  --num_train_epochs 2.0 \
+  --max_seq_length 384 \
+  --doc_stride 128 \
+  --output_dir /tmp/output_dir/
+```
+
+**Results**:
+
+{'exact': **67.38**, 'f1': **71.36**} 
@@ -0,0 +1,31 @@
+# Tensorflow CamemBERT
+
+In this repository you will find different versions of the CamemBERT model for Tensorflow.
+
+## CamemBERT
+
+[CamemBERT](https://camembert-model.fr/) is a state-of-the-art language model for French based on the RoBERTa architecture pretrained on the French subcorpus of the newly available multilingual corpus OSCAR.
+
+## Model Weights
+
+| Model                            | Downloads
+| -------------------------------- | ---------------------------------------------------------------------------------------------------------------
+| `jplu/tf-camembert-base`   | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-camembert-base/config.json) • [`tf_model.h5`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-camembert-base/tf_model.h5)
+
+## Usage
+
+With Transformers >= 2.4 the Tensorflow models of CamemBERT can be loaded like:
+
+```python
+from transformers import TFCamembertModel
+
+model = TFCamembertModel.from_pretrained("jplu/tf-camembert-base")
+```
+
+## Huggingface model hub
+
+All models are available on the [Huggingface model hub](https://huggingface.co/jplu).
+
+## Acknowledgments
+
+Thanks to all the Huggingface team for the support and their amazing library!
@@ -0,0 +1,36 @@
+# Tensorflow XLM-RoBERTa
+
+In this repository you will find different versions of the XLM-RoBERTa model for Tensorflow.
+
+## XLM-RoBERTa
+
+[XLM-RoBERTa](https://ai.facebook.com/blog/-xlm-r-state-of-the-art-cross-lingual-understanding-through-self-supervision/) is a scaled cross lingual sentence encoder. It is trained on 2.5T of data across 100 languages data filtered from Common Crawl. XLM-R achieves state-of-the-arts results on multiple cross lingual benchmarks.
+
+## Model Weights
+
+| Model                            | Downloads
+| -------------------------------- | ---------------------------------------------------------------------------------------------------------------
+| `jplu/tf-xlm-roberta-base`   | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-base/config.json) • [`tf_model.h5`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-base/tf_model.h5)
+| `jplu/tf-xlm-roberta-large`   | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-large/config.json) • [`tf_model.h5`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-large/tf_model.h5)
+
+## Usage
+
+With Transformers >= 2.4 the Tensorflow models of XLM-RoBERTa can be loaded like:
+
+```python
+from transformers import TFXLMRobertaModel
+
+model = TFXLMRobertaModel.from_pretrained("jplu/tf-xlm-roberta-base")
+```
+Or
+```
+model = TFXLMRobertaModel.from_pretrained("jplu/tf-xlm-roberta-large")
+```
+
+## Huggingface model hub
+
+All models are available on the [Huggingface model hub](https://huggingface.co/jplu).
+
+## Acknowledgments
+
+Thanks to all the Huggingface team for the support and their amazing library!
@@ -0,0 +1,36 @@
+# Tensorflow XLM-RoBERTa
+
+In this repository you will find different versions of the XLM-RoBERTa model for Tensorflow.
+
+## XLM-RoBERTa
+
+[XLM-RoBERTa](https://ai.facebook.com/blog/-xlm-r-state-of-the-art-cross-lingual-understanding-through-self-supervision/) is a scaled cross lingual sentence encoder. It is trained on 2.5T of data across 100 languages data filtered from Common Crawl. XLM-R achieves state-of-the-arts results on multiple cross lingual benchmarks.
+
+## Model Weights
+
+| Model                            | Downloads
+| -------------------------------- | ---------------------------------------------------------------------------------------------------------------
+| `jplu/tf-xlm-roberta-base`   | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-base/config.json) • [`tf_model.h5`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-base/tf_model.h5)
+| `jplu/tf-xlm-roberta-large`   | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-large/config.json) • [`tf_model.h5`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-large/tf_model.h5)
+
+## Usage
+
+With Transformers >= 2.4 the Tensorflow models of XLM-RoBERTa can be loaded like:
+
+```python
+from transformers import TFXLMRobertaModel
+
+model = TFXLMRobertaModel.from_pretrained("jplu/tf-xlm-roberta-base")
+```
+Or
+```
+model = TFXLMRobertaModel.from_pretrained("jplu/tf-xlm-roberta-large")
+```
+
+## Huggingface model hub
+
+All models are available on the [Huggingface model hub](https://huggingface.co/jplu).
+
+## Acknowledgments
+
+Thanks to all the Huggingface team for the support and their amazing library!
@@ -0,0 +1,25 @@
+## How to build a dummy model
+
+
+```python
+from transformers.configuration_bert import BertConfig
+from transformers.modeling_bert import BertForMaskedLM
+from transformers.modeling_tf_bert import TFBertForMaskedLM
+from transformers.tokenization_bert import BertTokenizer
+
+
+SMALL_MODEL_IDENTIFIER = "julien-c/bert-xsmall-dummy"
+DIRNAME = "./bert-xsmall-dummy"
+
+config = BertConfig(10, 20, 1, 1, 40)
+
+model = BertForMaskedLM(config)
+model.save_pretrained(DIRNAME)
+
+tf_model = TFBertForMaskedLM.from_pretrained(DIRNAME, from_pt=True)
+tf_model.save_pretrained(DIRNAME)
+
+# Slightly different for tokenizer.
+# tokenizer = BertTokenizer.from_pretrained(DIRNAME)
+# tokenizer.save_pretrained()
+```
@@ -0,0 +1,52 @@
+
+```python
+import json
+import os
+from transformers.configuration_roberta import RobertaConfig
+from transformers import RobertaForMaskedLM, TFRobertaForMaskedLM
+
+DIRNAME = "./dummy-unknown"
+
+
+config = RobertaConfig(10, 20, 1, 1, 40)
+
+model = RobertaForMaskedLM(config)
+model.save_pretrained(DIRNAME)
+
+tf_model = TFRobertaForMaskedLM.from_pretrained(DIRNAME, from_pt=True)
+tf_model.save_pretrained(DIRNAME)
+
+# Tokenizer:
+
+vocab = [
+    "l",
+    "o",
+    "w",
+    "e",
+    "r",
+    "s",
+    "t",
+    "i",
+    "d",
+    "n",
+    "\u0120",
+    "\u0120l",
+    "\u0120n",
+    "\u0120lo",
+    "\u0120low",
+    "er",
+    "\u0120lowest",
+    "\u0120newer",
+    "\u0120wider",
+    "<unk>",
+]
+vocab_tokens = dict(zip(vocab, range(len(vocab))))
+merges = ["#version: 0.2", "\u0120 l", "\u0120l o", "\u0120lo w", "e r", ""]
+
+vocab_file = os.path.join(DIRNAME, "vocab.json")
+merges_file = os.path.join(DIRNAME, "merges.txt")
+with open(vocab_file, "w", encoding="utf-8") as fp:
+    fp.write(json.dumps(vocab_tokens) + "\n")
+with open(merges_file, "w", encoding="utf-8") as fp:
+    fp.write("\n".join(merges))
+```
@@ -23,6 +23,8 @@ To create the package for pypi.

   twine upload dist/* -r pypitest
   (pypi suggest using twine as other methods upload files via plaintext.)
+   You may have to specify the repository url, use the following command then:
+   twine upload dist/* -r pypitest --repository-url=https://test.pypi.org/legacy/

   Check that you can install it in a virtualenv by running:
   pip install -i https://testpypi.python.org/pypi transformers
@@ -73,7 +75,7 @@ extras["dev"] = extras["testing"] + extras["quality"] + ["mecab-python3", "sciki

 setup(
    name="transformers",
-    version="2.4.0",
+    version="2.4.1",
    author="Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Google AI Language Team Authors, Open AI team Authors, Facebook AI Authors, Carnegie Mellon University Authors",
    author_email="thomas@huggingface.co",
    description="State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch",
@@ -2,7 +2,7 @@
 # There's no way to ignore "F401 '...' imported but unused" warnings in this
 # module, but to preserve other warnings. So, don't check this module at all.

-__version__ = "2.4.0"
+__version__ = "2.4.1"

 # Work around to update TensorFlow's absl.logging threshold which alters the
 # default Python logging output behavior when present.
@@ -0,0 +1,58 @@
+import platform
+from argparse import ArgumentParser
+
+from transformers import __version__ as version
+from transformers import is_tf_available, is_torch_available
+from transformers.commands import BaseTransformersCLICommand
+
+
+def info_command_factory(_):
+    return EnvironmentCommand()
+
+
+class EnvironmentCommand(BaseTransformersCLICommand):
+    @staticmethod
+    def register_subcommand(parser: ArgumentParser):
+        download_parser = parser.add_parser("env")
+        download_parser.set_defaults(func=info_command_factory)
+
+    def run(self):
+        pt_version = "not installed"
+        pt_cuda_available = "NA"
+        if is_torch_available():
+            import torch
+
+            pt_version = torch.__version__
+            pt_cuda_available = torch.cuda.is_available()
+
+        tf_version = "not installed"
+        tf_cuda_available = "NA"
+        if is_tf_available():
+            import tensorflow as tf
+
+            tf_version = tf.__version__
+            try:
+                # deprecated in v2.1
+                tf_cuda_available = tf.test.is_gpu_available()
+            except AttributeError:
+                # returns list of devices, convert to bool
+                tf_cuda_available = bool(tf.config.list_physical_devices("GPU"))
+
+        info = {
+            "`transformers` version": version,
+            "Platform": platform.platform(),
+            "Python version": platform.python_version(),
+            "PyTorch version (GPU?)": "{} ({})".format(pt_version, pt_cuda_available),
+            "Tensorflow version (GPU?)": "{} ({})".format(tf_version, tf_cuda_available),
+            "Using GPU in script?": "<fill in>",
+            "Using distributed or parallel set-up in script?": "<fill in>",
+        }
+
+        print("\nCopy-and-paste the text below in your GitHub issue and FILL OUT the two last points.\n")
+        print(self.format_dict(info))
+
+        return info
+
+    @staticmethod
+    def format_dict(d):
+        return "\n".join(["- {}: {}".format(prop, val) for prop, val in d.items()]) + "\n"
@@ -50,8 +50,8 @@ class FlaubertConfig(XLMConfig):
                Probability to drop layers during training (Fan et al., Reducing Transformer Depth on Demand
                with Structured Dropout. ICLR 2020)
            vocab_size (:obj:`int`, optional, defaults to 30145):
-                Vocabulary size of the XLM model. Defines the different tokens that
-                can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.XLMModel`.
+                Vocabulary size of the Flaubert model. Defines the different tokens that
+                can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.FlaubertModel`.
            emb_dim (:obj:`int`, optional, defaults to 2048):
                Dimensionality of the encoder layers and the pooler layer.
            n_layer (:obj:`int`, optional, defaults to 12):
@@ -129,8 +129,8 @@ ALL_PRETRAINED_MODEL_ARCHIVE_MAP = dict(
        ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
        CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
        T5_PRETRAINED_MODEL_ARCHIVE_MAP,
-        XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
        FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
+        XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
    ]
    for key, value, in pretrained_map.items()
 )
@@ -148,9 +148,9 @@ MODEL_MAPPING = OrderedDict(
        (GPT2Config, GPT2Model),
        (TransfoXLConfig, TransfoXLModel),
        (XLNetConfig, XLNetModel),
+        (FlaubertConfig, FlaubertModel),
        (XLMConfig, XLMModel),
        (CTRLConfig, CTRLModel),
-        (FlaubertConfig, FlaubertModel),
    ]
 )

@@ -167,9 +167,9 @@ MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
        (GPT2Config, GPT2LMHeadModel),
        (TransfoXLConfig, TransfoXLLMHeadModel),
        (XLNetConfig, XLNetLMHeadModel),
+        (FlaubertConfig, FlaubertWithLMHeadModel),
        (XLMConfig, XLMWithLMHeadModel),
        (CTRLConfig, CTRLLMHeadModel),
-        (FlaubertConfig, FlaubertWithLMHeadModel),
    ]
 )

@@ -186,9 +186,9 @@ MODEL_WITH_LM_HEAD_MAPPING = OrderedDict(
        (GPT2Config, GPT2LMHeadModel),
        (TransfoXLConfig, TransfoXLLMHeadModel),
        (XLNetConfig, XLNetLMHeadModel),
+        (FlaubertConfig, FlaubertWithLMHeadModel),
        (XLMConfig, XLMWithLMHeadModel),
        (CTRLConfig, CTRLLMHeadModel),
-        (FlaubertConfig, FlaubertWithLMHeadModel),
    ]
 )

@@ -201,8 +201,8 @@ MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING = OrderedDict(
        (RobertaConfig, RobertaForSequenceClassification),
        (BertConfig, BertForSequenceClassification),
        (XLNetConfig, XLNetForSequenceClassification),
-        (XLMConfig, XLMForSequenceClassification),
        (FlaubertConfig, FlaubertForSequenceClassification),
+        (XLMConfig, XLMForSequenceClassification),
    ]
 )

@@ -213,8 +213,8 @@ MODEL_FOR_QUESTION_ANSWERING_MAPPING = OrderedDict(
        (RobertaConfig, RobertaForQuestionAnswering),
        (BertConfig, BertForQuestionAnswering),
        (XLNetConfig, XLNetForQuestionAnswering),
-        (XLMConfig, XLMForQuestionAnswering),
        (FlaubertConfig, FlaubertForQuestionAnswering),
+        (XLMConfig, XLMForQuestionAnswering),
    ]
 )

@@ -232,7 +232,7 @@ class PreTrainedEncoderDecoder(nn.Module):
            encoder_outputs = ()

        kwargs_decoder["encoder_hidden_states"] = encoder_hidden_states
-        decoder_outputs = self.decoder(decoder_input_ids, encoder_hidden_states, **kwargs_decoder)
+        decoder_outputs = self.decoder(decoder_input_ids, **kwargs_decoder)

        return decoder_outputs + encoder_outputs

@@ -586,7 +586,7 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin):
        self,
        input_ids=None,
        max_length=None,
-        do_sample=None,
+        do_sample=True,
        num_beams=None,
        temperature=None,
        top_k=None,
@@ -617,7 +617,7 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin):
                The max length of the sequence to be generated.  Between 1 and infinity. Default to 20.

            do_sample: (`optional`) bool
-                If set to `False` greedy decoding is used. Otherwise sampling is used. Default to greedy sampling.
+                If set to `False` greedy decoding is used. Otherwise sampling is used. Defaults to `True`.

            num_beams: (`optional`) int
                Number of beams for beam search. Must be between 1 and infinity. 1 means no beam search. Default to 1.
@@ -326,7 +326,7 @@ class Pipeline(_ScikitCompat):
        self,
        model,
        tokenizer: PreTrainedTokenizer = None,
-        modelcard: ModelCard = None,
+        modelcard: Optional[ModelCard] = None,
        framework: Optional[str] = None,
        args_parser: ArgumentHandler = None,
        device: int = -1,
@@ -358,7 +358,8 @@ class Pipeline(_ScikitCompat):

        self.model.save_pretrained(save_directory)
        self.tokenizer.save_pretrained(save_directory)
-        self.modelcard.save_pretrained(save_directory)
+        if self.modelcard is not None:
+            self.modelcard.save_pretrained(save_directory)

    def transform(self, X):
        """
@@ -476,7 +477,7 @@ class FeatureExtractionPipeline(Pipeline):
        self,
        model,
        tokenizer: PreTrainedTokenizer = None,
-        modelcard: ModelCard = None,
+        modelcard: Optional[ModelCard] = None,
        framework: Optional[str] = None,
        args_parser: ArgumentHandler = None,
        device: int = -1,
@@ -515,7 +516,7 @@ class FillMaskPipeline(Pipeline):
        self,
        model,
        tokenizer: PreTrainedTokenizer = None,
-        modelcard: ModelCard = None,
+        modelcard: Optional[ModelCard] = None,
        framework: Optional[str] = None,
        args_parser: ArgumentHandler = None,
        device: int = -1,
@@ -582,7 +583,7 @@ class NerPipeline(Pipeline):
        self,
        model,
        tokenizer: PreTrainedTokenizer = None,
-        modelcard: ModelCard = None,
+        modelcard: Optional[ModelCard] = None,
        framework: Optional[str] = None,
        args_parser: ArgumentHandler = None,
        device: int = -1,
@@ -721,7 +722,7 @@ class QuestionAnsweringPipeline(Pipeline):
        self,
        model,
        tokenizer: Optional[PreTrainedTokenizer],
-        modelcard: Optional[ModelCard],
+        modelcard: Optional[ModelCard] = None,
        framework: Optional[str] = None,
        device: int = -1,
        **kwargs
@@ -25,6 +25,7 @@ from .configuration_auto import (
    CamembertConfig,
    CTRLConfig,
    DistilBertConfig,
+    FlaubertConfig,
    GPT2Config,
    OpenAIGPTConfig,
    RobertaConfig,
@@ -41,6 +42,7 @@ from .tokenization_bert_japanese import BertJapaneseTokenizer
 from .tokenization_camembert import CamembertTokenizer
 from .tokenization_ctrl import CTRLTokenizer
 from .tokenization_distilbert import DistilBertTokenizer
+from .tokenization_flaubert import FlaubertTokenizer
 from .tokenization_gpt2 import GPT2Tokenizer
 from .tokenization_openai import OpenAIGPTTokenizer
 from .tokenization_roberta import RobertaTokenizer
@@ -67,6 +69,7 @@ TOKENIZER_MAPPING = OrderedDict(
        (GPT2Config, GPT2Tokenizer),
        (TransfoXLConfig, TransfoXLTokenizer),
        (XLNetConfig, XLNetTokenizer),
+        (FlaubertConfig, FlaubertTokenizer),
        (XLMConfig, XLMTokenizer),
        (CTRLConfig, CTRLTokenizer),
    ]
@@ -39,6 +39,14 @@ if is_torch_available():
        BertForQuestionAnswering,
    )
    from transformers.modeling_bert import BERT_PRETRAINED_MODEL_ARCHIVE_MAP
+    from transformers.modeling_auto import (
+        MODEL_MAPPING,
+        MODEL_FOR_PRETRAINING_MAPPING,
+        MODEL_FOR_QUESTION_ANSWERING_MAPPING,
+        MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING,
+        MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING,
+        MODEL_WITH_LM_HEAD_MAPPING,
+    )


@require_torch
@@ -127,3 +135,26 @@ class AutoModelTest(unittest.TestCase):
        self.assertIsInstance(model, RobertaForMaskedLM)
        self.assertEqual(model.num_parameters(), 14830)
        self.assertEqual(model.num_parameters(only_trainable=True), 14830)
+
+    def test_parents_and_children_in_mappings(self):
+        # Test that the children are placed before the parents in the mappings, as the `instanceof` will be triggered
+        # by the parents and will return the wrong configuration type when using auto models
+
+        mappings = (
+            MODEL_MAPPING,
+            MODEL_FOR_PRETRAINING_MAPPING,
+            MODEL_FOR_QUESTION_ANSWERING_MAPPING,
+            MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING,
+            MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING,
+            MODEL_WITH_LM_HEAD_MAPPING,
+        )
+
+        for mapping in mappings:
+            mapping = tuple(mapping.items())
+            for index, (child_config, child_model) in enumerate(mapping[1:]):
+                for parent_config, parent_model in mapping[: index + 1]:
+                    with self.subTest(
+                        msg="Testing if {} is child of {}".format(child_config.__name__, parent_config.__name__)
+                    ):
+                        self.assertFalse(issubclass(child_config, parent_config))
+                        self.assertFalse(issubclass(child_model, parent_model))
@@ -117,23 +117,11 @@ class ModelTesterMixin:

    def test_attention_outputs(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
-
-        decoder_seq_length = (
-            self.model_tester.decoder_seq_length
-            if hasattr(self.model_tester, "decoder_seq_length")
-            else self.model_tester.seq_length
-        )
-        encoder_seq_length = (
-            self.model_tester.encoder_seq_length
-            if hasattr(self.model_tester, "encoder_seq_length")
-            else self.model_tester.seq_length
-        )
-        decoder_key_length = (
-            self.model_tester.key_length if hasattr(self.model_tester, "key_length") else decoder_seq_length
-        )
-        encoder_key_length = (
-            self.model_tester.key_length if hasattr(self.model_tester, "key_length") else encoder_seq_length
-        )
+        seq_len = getattr(self.model_tester, "seq_length", None)
+        decoder_seq_length = getattr(self.model_tester, "decoder_seq_length", seq_len)
+        encoder_seq_length = getattr(self.model_tester, "encoder_seq_length", seq_len)
+        decoder_key_length = getattr(self.model_tester, "key_length", decoder_seq_length)
+        encoder_key_length = getattr(self.model_tester, "key_length", encoder_seq_length)

        for model_class in self.all_model_classes:
            config.output_attentions = True
@@ -0,0 +1,392 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import unittest
+
+from transformers import is_torch_available
+
+from .test_configuration_common import ConfigTester
+from .test_modeling_common import ModelTesterMixin, ids_tensor
+from .utils import CACHE_DIR, require_torch, slow, torch_device
+
+
+if is_torch_available():
+    from transformers import (
+        FlaubertConfig,
+        FlaubertModel,
+        FlaubertWithLMHeadModel,
+        FlaubertForQuestionAnswering,
+        FlaubertForQuestionAnsweringSimple,
+        FlaubertForSequenceClassification,
+    )
+    from transformers.modeling_flaubert import FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
+
+
+@require_torch
+class FlaubertModelTest(ModelTesterMixin, unittest.TestCase):
+
+    all_model_classes = (
+        (
+            FlaubertModel,
+            FlaubertWithLMHeadModel,
+            FlaubertForQuestionAnswering,
+            FlaubertForQuestionAnsweringSimple,
+            FlaubertForSequenceClassification,
+        )
+        if is_torch_available()
+        else ()
+    )
+
+    class FlaubertModelTester(object):
+        def __init__(
+            self,
+            parent,
+            batch_size=13,
+            seq_length=7,
+            is_training=True,
+            use_input_lengths=True,
+            use_token_type_ids=True,
+            use_labels=True,
+            gelu_activation=True,
+            sinusoidal_embeddings=False,
+            causal=False,
+            asm=False,
+            n_langs=2,
+            vocab_size=99,
+            n_special=0,
+            hidden_size=32,
+            num_hidden_layers=5,
+            num_attention_heads=4,
+            hidden_dropout_prob=0.1,
+            attention_probs_dropout_prob=0.1,
+            max_position_embeddings=512,
+            type_vocab_size=16,
+            type_sequence_label_size=2,
+            initializer_range=0.02,
+            num_labels=3,
+            num_choices=4,
+            summary_type="last",
+            use_proj=True,
+            scope=None,
+        ):
+            self.parent = parent
+            self.batch_size = batch_size
+            self.seq_length = seq_length
+            self.is_training = is_training
+            self.use_input_lengths = use_input_lengths
+            self.use_token_type_ids = use_token_type_ids
+            self.use_labels = use_labels
+            self.gelu_activation = gelu_activation
+            self.sinusoidal_embeddings = sinusoidal_embeddings
+            self.asm = asm
+            self.n_langs = n_langs
+            self.vocab_size = vocab_size
+            self.n_special = n_special
+            self.summary_type = summary_type
+            self.causal = causal
+            self.use_proj = use_proj
+            self.hidden_size = hidden_size
+            self.num_hidden_layers = num_hidden_layers
+            self.num_attention_heads = num_attention_heads
+            self.hidden_dropout_prob = hidden_dropout_prob
+            self.attention_probs_dropout_prob = attention_probs_dropout_prob
+            self.max_position_embeddings = max_position_embeddings
+            self.n_langs = n_langs
+            self.type_sequence_label_size = type_sequence_label_size
+            self.initializer_range = initializer_range
+            self.summary_type = summary_type
+            self.num_labels = num_labels
+            self.num_choices = num_choices
+            self.scope = scope
+
+        def prepare_config_and_inputs(self):
+            input_ids = ids_tensor([self.batch_size, self.seq_length], self.vocab_size)
+            input_mask = ids_tensor([self.batch_size, self.seq_length], 2).float()
+
+            input_lengths = None
+            if self.use_input_lengths:
+                input_lengths = (
+                    ids_tensor([self.batch_size], vocab_size=2) + self.seq_length - 2
+                )  # small variation of seq_length
+
+            token_type_ids = None
+            if self.use_token_type_ids:
+                token_type_ids = ids_tensor([self.batch_size, self.seq_length], self.n_langs)
+
+            sequence_labels = None
+            token_labels = None
+            is_impossible_labels = None
+            if self.use_labels:
+                sequence_labels = ids_tensor([self.batch_size], self.type_sequence_label_size)
+                token_labels = ids_tensor([self.batch_size, self.seq_length], self.num_labels)
+                is_impossible_labels = ids_tensor([self.batch_size], 2).float()
+
+            config = FlaubertConfig(
+                vocab_size=self.vocab_size,
+                n_special=self.n_special,
+                emb_dim=self.hidden_size,
+                n_layers=self.num_hidden_layers,
+                n_heads=self.num_attention_heads,
+                dropout=self.hidden_dropout_prob,
+                attention_dropout=self.attention_probs_dropout_prob,
+                gelu_activation=self.gelu_activation,
+                sinusoidal_embeddings=self.sinusoidal_embeddings,
+                asm=self.asm,
+                causal=self.causal,
+                n_langs=self.n_langs,
+                max_position_embeddings=self.max_position_embeddings,
+                initializer_range=self.initializer_range,
+                summary_type=self.summary_type,
+                use_proj=self.use_proj,
+            )
+
+            return (
+                config,
+                input_ids,
+                token_type_ids,
+                input_lengths,
+                sequence_labels,
+                token_labels,
+                is_impossible_labels,
+                input_mask,
+            )
+
+        def check_loss_output(self, result):
+            self.parent.assertListEqual(list(result["loss"].size()), [])
+
+        def create_and_check_flaubert_model(
+            self,
+            config,
+            input_ids,
+            token_type_ids,
+            input_lengths,
+            sequence_labels,
+            token_labels,
+            is_impossible_labels,
+            input_mask,
+        ):
+            model = FlaubertModel(config=config)
+            model.to(torch_device)
+            model.eval()
+            outputs = model(input_ids, lengths=input_lengths, langs=token_type_ids)
+            outputs = model(input_ids, langs=token_type_ids)
+            outputs = model(input_ids)
+            sequence_output = outputs[0]
+            result = {
+                "sequence_output": sequence_output,
+            }
+            self.parent.assertListEqual(
+                list(result["sequence_output"].size()), [self.batch_size, self.seq_length, self.hidden_size]
+            )
+
+        def create_and_check_flaubert_lm_head(
+            self,
+            config,
+            input_ids,
+            token_type_ids,
+            input_lengths,
+            sequence_labels,
+            token_labels,
+            is_impossible_labels,
+            input_mask,
+        ):
+            model = FlaubertWithLMHeadModel(config)
+            model.to(torch_device)
+            model.eval()
+
+            loss, logits = model(input_ids, token_type_ids=token_type_ids, labels=token_labels)
+
+            result = {
+                "loss": loss,
+                "logits": logits,
+            }
+
+            self.parent.assertListEqual(list(result["loss"].size()), [])
+            self.parent.assertListEqual(
+                list(result["logits"].size()), [self.batch_size, self.seq_length, self.vocab_size]
+            )
+
+        def create_and_check_flaubert_simple_qa(
+            self,
+            config,
+            input_ids,
+            token_type_ids,
+            input_lengths,
+            sequence_labels,
+            token_labels,
+            is_impossible_labels,
+            input_mask,
+        ):
+            model = FlaubertForQuestionAnsweringSimple(config)
+            model.to(torch_device)
+            model.eval()
+
+            outputs = model(input_ids)
+
+            outputs = model(input_ids, start_positions=sequence_labels, end_positions=sequence_labels)
+            loss, start_logits, end_logits = outputs
+
+            result = {
+                "loss": loss,
+                "start_logits": start_logits,
+                "end_logits": end_logits,
+            }
+            self.parent.assertListEqual(list(result["start_logits"].size()), [self.batch_size, self.seq_length])
+            self.parent.assertListEqual(list(result["end_logits"].size()), [self.batch_size, self.seq_length])
+            self.check_loss_output(result)
+
+        def create_and_check_flaubert_qa(
+            self,
+            config,
+            input_ids,
+            token_type_ids,
+            input_lengths,
+            sequence_labels,
+            token_labels,
+            is_impossible_labels,
+            input_mask,
+        ):
+            model = FlaubertForQuestionAnswering(config)
+            model.to(torch_device)
+            model.eval()
+
+            outputs = model(input_ids)
+            start_top_log_probs, start_top_index, end_top_log_probs, end_top_index, cls_logits = outputs
+
+            outputs = model(
+                input_ids,
+                start_positions=sequence_labels,
+                end_positions=sequence_labels,
+                cls_index=sequence_labels,
+                is_impossible=is_impossible_labels,
+                p_mask=input_mask,
+            )
+
+            outputs = model(
+                input_ids,
+                start_positions=sequence_labels,
+                end_positions=sequence_labels,
+                cls_index=sequence_labels,
+                is_impossible=is_impossible_labels,
+            )
+
+            (total_loss,) = outputs
+
+            outputs = model(input_ids, start_positions=sequence_labels, end_positions=sequence_labels)
+
+            (total_loss,) = outputs
+
+            result = {
+                "loss": total_loss,
+                "start_top_log_probs": start_top_log_probs,
+                "start_top_index": start_top_index,
+                "end_top_log_probs": end_top_log_probs,
+                "end_top_index": end_top_index,
+                "cls_logits": cls_logits,
+            }
+
+            self.parent.assertListEqual(list(result["loss"].size()), [])
+            self.parent.assertListEqual(
+                list(result["start_top_log_probs"].size()), [self.batch_size, model.config.start_n_top]
+            )
+            self.parent.assertListEqual(
+                list(result["start_top_index"].size()), [self.batch_size, model.config.start_n_top]
+            )
+            self.parent.assertListEqual(
+                list(result["end_top_log_probs"].size()),
+                [self.batch_size, model.config.start_n_top * model.config.end_n_top],
+            )
+            self.parent.assertListEqual(
+                list(result["end_top_index"].size()),
+                [self.batch_size, model.config.start_n_top * model.config.end_n_top],
+            )
+            self.parent.assertListEqual(list(result["cls_logits"].size()), [self.batch_size])
+
+        def create_and_check_flaubert_sequence_classif(
+            self,
+            config,
+            input_ids,
+            token_type_ids,
+            input_lengths,
+            sequence_labels,
+            token_labels,
+            is_impossible_labels,
+            input_mask,
+        ):
+            model = FlaubertForSequenceClassification(config)
+            model.to(torch_device)
+            model.eval()
+
+            (logits,) = model(input_ids)
+            loss, logits = model(input_ids, labels=sequence_labels)
+
+            result = {
+                "loss": loss,
+                "logits": logits,
+            }
+
+            self.parent.assertListEqual(list(result["loss"].size()), [])
+            self.parent.assertListEqual(
+                list(result["logits"].size()), [self.batch_size, self.type_sequence_label_size]
+            )
+
+        def prepare_config_and_inputs_for_common(self):
+            config_and_inputs = self.prepare_config_and_inputs()
+            (
+                config,
+                input_ids,
+                token_type_ids,
+                input_lengths,
+                sequence_labels,
+                token_labels,
+                is_impossible_labels,
+                input_mask,
+            ) = config_and_inputs
+            inputs_dict = {"input_ids": input_ids, "token_type_ids": token_type_ids, "lengths": input_lengths}
+            return config, inputs_dict
+
+    def setUp(self):
+        self.model_tester = FlaubertModelTest.FlaubertModelTester(self)
+        self.config_tester = ConfigTester(self, config_class=FlaubertConfig, emb_dim=37)
+
+    def test_config(self):
+        self.config_tester.run_common_tests()
+
+    def test_flaubert_model(self):
+        config_and_inputs = self.model_tester.prepare_config_and_inputs()
+        self.model_tester.create_and_check_flaubert_model(*config_and_inputs)
+
+    def test_flaubert_lm_head(self):
+        config_and_inputs = self.model_tester.prepare_config_and_inputs()
+        self.model_tester.create_and_check_flaubert_lm_head(*config_and_inputs)
+
+    def test_flaubert_simple_qa(self):
+        config_and_inputs = self.model_tester.prepare_config_and_inputs()
+        self.model_tester.create_and_check_flaubert_simple_qa(*config_and_inputs)
+
+    def test_flaubert_qa(self):
+        config_and_inputs = self.model_tester.prepare_config_and_inputs()
+        self.model_tester.create_and_check_flaubert_qa(*config_and_inputs)
+
+    def test_flaubert_sequence_classif(self):
+        config_and_inputs = self.model_tester.prepare_config_and_inputs()
+        self.model_tester.create_and_check_flaubert_sequence_classif(*config_and_inputs)
+
+    @slow
+    def test_model_from_pretrained(self):
+        for model_name in list(FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
+            model = FlaubertModel.from_pretrained(model_name, cache_dir=CACHE_DIR)
+            self.assertIsNotNone(model)
@@ -32,7 +32,7 @@ if is_torch_available():
        RobertaForSequenceClassification,
        RobertaForTokenClassification,
    )
-    from transformers.modeling_roberta import RobertaEmbeddings
+    from transformers.modeling_roberta import RobertaEmbeddings, RobertaForMultipleChoice, RobertaForQuestionAnswering
    from transformers.modeling_roberta import ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP


@@ -184,6 +184,51 @@ class RobertaModelTest(ModelTesterMixin, unittest.TestCase):
            )
            self.check_loss_output(result)

+        def create_and_check_roberta_for_multiple_choice(
+            self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels, choice_labels
+        ):
+            config.num_choices = self.num_choices
+            model = RobertaForMultipleChoice(config=config)
+            model.to(torch_device)
+            model.eval()
+            multiple_choice_inputs_ids = input_ids.unsqueeze(1).expand(-1, self.num_choices, -1).contiguous()
+            multiple_choice_token_type_ids = token_type_ids.unsqueeze(1).expand(-1, self.num_choices, -1).contiguous()
+            multiple_choice_input_mask = input_mask.unsqueeze(1).expand(-1, self.num_choices, -1).contiguous()
+            loss, logits = model(
+                multiple_choice_inputs_ids,
+                attention_mask=multiple_choice_input_mask,
+                token_type_ids=multiple_choice_token_type_ids,
+                labels=choice_labels,
+            )
+            result = {
+                "loss": loss,
+                "logits": logits,
+            }
+            self.parent.assertListEqual(list(result["logits"].size()), [self.batch_size, self.num_choices])
+            self.check_loss_output(result)
+
+        def create_and_check_roberta_for_question_answering(
+            self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels, choice_labels
+        ):
+            model = RobertaForQuestionAnswering(config=config)
+            model.to(torch_device)
+            model.eval()
+            loss, start_logits, end_logits = model(
+                input_ids,
+                attention_mask=input_mask,
+                token_type_ids=token_type_ids,
+                start_positions=sequence_labels,
+                end_positions=sequence_labels,
+            )
+            result = {
+                "loss": loss,
+                "start_logits": start_logits,
+                "end_logits": end_logits,
+            }
+            self.parent.assertListEqual(list(result["start_logits"].size()), [self.batch_size, self.seq_length])
+            self.parent.assertListEqual(list(result["end_logits"].size()), [self.batch_size, self.seq_length])
+            self.check_loss_output(result)
+
        def prepare_config_and_inputs_for_common(self):
            config_and_inputs = self.prepare_config_and_inputs()
            (
@@ -213,6 +258,18 @@ class RobertaModelTest(ModelTesterMixin, unittest.TestCase):
        config_and_inputs = self.model_tester.prepare_config_and_inputs()
        self.model_tester.create_and_check_roberta_for_masked_lm(*config_and_inputs)

+    def test_for_token_classification(self):
+        config_and_inputs = self.model_tester.prepare_config_and_inputs()
+        self.model_tester.create_and_check_roberta_for_token_classification(*config_and_inputs)
+
+    def test_for_multiple_choice(self):
+        config_and_inputs = self.model_tester.prepare_config_and_inputs()
+        self.model_tester.create_and_check_roberta_for_multiple_choice(*config_and_inputs)
+
+    def test_for_question_answering(self):
+        config_and_inputs = self.model_tester.prepare_config_and_inputs()
+        self.model_tester.create_and_check_roberta_for_question_answering(*config_and_inputs)
+
    @slow
    def test_model_from_pretrained(self):
        for model_name in list(ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
@@ -200,6 +200,10 @@ class TFRobertaModelTest(TFModelTesterMixin, unittest.TestCase):
        config_and_inputs = self.model_tester.prepare_config_and_inputs()
        self.model_tester.create_and_check_roberta_for_masked_lm(*config_and_inputs)

+    def test_for_token_classification(self):
+        config_and_inputs = self.model_tester.prepare_config_and_inputs()
+        self.model_tester.create_and_check_roberta_for_token_classification(*config_and_inputs)
+
    @slow
    def test_model_from_pretrained(self):
        for model_name in list(TF_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
@@ -25,6 +25,7 @@ from transformers import (
    GPT2Tokenizer,
    RobertaTokenizer,
 )
+from transformers.tokenization_auto import TOKENIZER_MAPPING

 from .utils import DUMMY_UNKWOWN_IDENTIFIER, SMALL_MODEL_IDENTIFIER, slow  # noqa: F401

@@ -70,3 +71,19 @@ class AutoTokenizerTest(unittest.TestCase):
        for tokenizer_class in [BertTokenizer, AutoTokenizer]:
            with self.assertRaises(EnvironmentError):
                _ = tokenizer_class.from_pretrained("julien-c/herlolip-not-exists")
+
+    def test_parents_and_children_in_mappings(self):
+        # Test that the children are placed before the parents in the mappings, as the `instanceof` will be triggered
+        # by the parents and will return the wrong configuration type when using auto models
+
+        mappings = (TOKENIZER_MAPPING,)
+
+        for mapping in mappings:
+            mapping = tuple(mapping.items())
+            for index, (child_config, child_model) in enumerate(mapping[1:]):
+                for parent_config, parent_model in mapping[: index + 1]:
+                    with self.subTest(
+                        msg="Testing if {} is child of {}".format(child_config.__name__, parent_config.__name__)
+                    ):
+                        self.assertFalse(issubclass(child_config, parent_config))
+                        self.assertFalse(issubclass(child_model, parent_model))
@@ -1,11 +1,12 @@
 #!/usr/bin/env python
 from argparse import ArgumentParser

-from transformers.commands.download import DownloadCommand
-from transformers.commands.run import RunCommand
-from transformers.commands.user import UserCommands
 from transformers.commands.convert import ConvertCommand
+from transformers.commands.download import DownloadCommand
+from transformers.commands.env import EnvironmentCommand
+from transformers.commands.run import RunCommand
 from transformers.commands.serving import ServeCommand
+from transformers.commands.user import UserCommands

 if __name__ == '__main__':
    parser = ArgumentParser('Transformers CLI tool', usage='transformers-cli <command> [<args>]')
@@ -14,6 +15,7 @@ if __name__ == '__main__':
    # Register commands
    ConvertCommand.register_subcommand(commands_parser)
    DownloadCommand.register_subcommand(commands_parser)
+    EnvironmentCommand.register_subcommand(commands_parser)
    RunCommand.register_subcommand(commands_parser)
    ServeCommand.register_subcommand(commands_parser)
    UserCommands.register_subcommand(commands_parser)
Author	SHA1	Message	Date
Lysandre	7ab2fe4471	RoBERTa TensorFlow Tests	2020-02-04 18:05:22 -05:00
Lysandre	23d03e4f60	Correct slow test	2020-02-04 18:05:22 -05:00
Lysandre	730d5e429b	Style	2020-02-04 18:05:22 -05:00
Lysandre	3d169f94fe	Flaubert PyTorch tests	2020-02-04 18:05:22 -05:00
Lysandre	28960e41fe	RoBERTa Pytorch tests	2020-02-04 18:05:22 -05:00
Yuval Pinter	d1ab1fab1b	pass langs parameter to certain XLM models (#2734 ) * pass langs parameter to certain XLM models Adding an argument that specifies the language the SQuAD dataset is in so language-sensitive XLMs (e.g. `xlm-mlm-tlm-xnli15-1024`) don't default to language `0`. Allows resolution of issue #1799 . * fixing from `make style` * fixing style (again)	2020-02-04 17:12:42 -05:00
sshleifer	9e5b549b4d	fix default getattr	2020-02-04 16:38:52 -05:00
sshleifer	25848a6094	double quotes	2020-02-04 16:38:52 -05:00
sshleifer	cbcb83f21d	minor cleanup of test_attention_outputs	2020-02-04 16:38:52 -05:00
Lysandre	3bf5417258	Revert erroneous fix	2020-02-04 16:31:07 -05:00
Lysandre	1ebfeb7946	Cast to long when masking tokens	2020-02-04 15:56:16 -05:00
Lysandre	9c67196b83	Update quickstart	2020-02-04 11:11:37 -05:00
Lysandre	90ab15cb7a	Remove redundant hidden states	2020-02-04 10:59:32 -05:00
Julien Chaumond	9a50828b5c	Pipelines: fix crash when modelcard is None cc @mfuntowicz does this seem correct?	2020-02-03 17:53:39 -05:00
Lysandre	6c1b23554f	Sample instead of greedy decoding by default in generate	2020-02-03 17:23:53 -05:00
Lysandre	239dd23f64	[Follow up 213] Masked indices should have -1 and not -100. Updating documentation + scripts that were forgotten	2020-02-03 16:08:05 -05:00
Martin Malmsten	522c5b5533	Added README.md to Swedish BERT models from National Library of Sweden	2020-02-03 09:09:34 -05:00
Julien Plu	9329e59700	Add READMEs to Tensorflow versions of CamemBERT and XLM-RoBERTa	2020-02-03 09:04:34 -05:00
Antonio Carlos Falcão Petri	2ba147ecff	Fix typo in examples/utils_ner.py "%s-%d".format() -> "{}-{}".format()	2020-02-01 11:10:57 -05:00
Bram Vanroy	9773e5e0d9	CLI script to gather environment info (#2699 ) * add "info" command to CLI As a convenience, add the info directive to CLI. Running `python transformers-cli info` will return a string containing the transformers version, platform, python version, PT/TF version and GPU support * Swap f-strings for .format Still supporting 3.5 so can't use f-strings (sad face) * Add reference in issue to CLI * Add the expected fields to issue template This way, people can still add the information manually if they want. (Though I fear they'll just ignore it.) * Remove heading from output * black-ify * order of imports Should ensure isort test passes * use is_X_available over import..pass * style * fix copy-paste bug * Rename command info -> env Also adds the command to CONTRIBUTING.md in "Did you find a bug" section	2020-02-01 10:38:14 -05:00
Julien ChaumondandStefan Schweter	ddb6f9476b	[model_cards] dbmdz models Co-Authored-By: Stefan Schweter <stefan-it@users.noreply.github.com>	2020-01-31 18:39:09 -05:00
Julien ChaumondandHenrykBorzymowski	6636826f04	[model_cards] Multilingual + Dutch SQuAD2.0 Co-Authored-By: HenrykBorzymowski <henrykborzymowski@users.noreply.github.com>	2020-01-31 18:39:09 -05:00
Julien Chaumond Loreto Parisi Simone Francia	98dadc98e1	[model_cards] UmBERTo Co-Authored-By: Loreto Parisi <loretoparisi@gmail.com> Co-Authored-By: Simone Francia <francia.simone1@gmail.com>	2020-01-31 18:39:09 -05:00
Julien Chaumond	d6fc34b459	[model_cards] add mine	2020-01-31 18:39:09 -05:00
Lysandre	d426b58b9e	Patch: v2.4.1	2020-01-31 14:55:33 -05:00
Lysandre	1e82cd8457	Flaubert auto tokenizer + tests cc @julien-c	2020-01-31 14:16:52 -05:00
Lysandre	d18d47be67	run_generation style	2020-01-31 12:05:48 -05:00
Lysandre	ff6f1492e8	FlauBERT load in AutoModel The FlauBERT configuration file inherits from XLMConfig, and is recognized as such when loading from AutoModels as the XLMConfig is checked before the FlaubertConfig. Changing the order solves this problem, but a test should be added.	2020-01-31 12:05:15 -05:00
Lysandre	7365f01d43	do_sample should be set to True in run_generation.py	2020-01-31 11:49:32 -05:00
Arnaud	3a21d6da6b	Typo on markdown link in README.md	2020-01-31 10:58:49 -05:00
Lysandre	0aa40e9569	v2.4.0 documentation	2020-01-31 09:55:34 -05:00
Lysandre	8036ceb7c5	Update commands for pypi test	2020-01-31 09:48:15 -05:00