refactor

refactor prepare inputs for Bert
make style
2020-04-20 19:41:08 +02:00 · 2020-04-20 19:38:38 +02:00 · 2020-04-20 19:22:23 +02:00 · 2020-04-20 19:02:49 +02:00 · 2020-04-20 18:44:33 +02:00 · 2020-04-20 18:44:15 +02:00
204 changed files with 10019 additions and 2646 deletions
@@ -66,6 +66,16 @@ jobs:
            - run: sudo pip install .[sklearn,torch,testing]
            - run: sudo pip install -r examples/requirements.txt
            - run: python -m pytest -n 8 --dist=loadfile -s -v ./examples/
+    build_doc:
+        working_directory: ~/transformers
+        docker:
+            - image: circleci/python:3.6
+        steps:
+            - checkout
+            - run: sudo pip install .[tf,torch,docs]
+            - run: cd docs && make html
+            - store_artifacts:
+                path: ./docs/_build
    deploy_doc:
        working_directory: ~/transformers
        docker:
@@ -117,4 +127,5 @@ workflows:
            - run_tests_torch_and_tf
            - run_tests_torch
            - run_tests_tf
+            - build_doc
            - deploy_doc: *workflow_filters
@@ -40,7 +40,7 @@ Steps to reproduce the behavior:
 <!-- A clear and concise description of what you would expect to happen. -->

 ## Environment info
-<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
+<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
 - `transformers` version:
@@ -11,9 +11,9 @@ jobs:
      uses: actions/setup-python@v1
      with:
        python-version: 3.7
-    - name: Install dependencies
-      run: |
-        pip install .[tf,torch,quality]
+    # - name: Install dependencies
+    #   run: |
+    #     pip install .[tf,torch,quality]



@@ -148,24 +148,26 @@ At some point in the future, you'll be able to seamlessly move from pre-training

 🤗 Transformers currently provides the following NLU/NLG architectures:

-1. **[BERT](https://github.com/google-research/bert)** (from Google) released with the paper [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.
-2. **[GPT](https://github.com/openai/finetune-transformer-lm)** (from OpenAI) released with the paper [Improving Language Understanding by Generative Pre-Training](https://blog.openai.com/language-unsupervised/) by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.
-3. **[GPT-2](https://blog.openai.com/better-language-models/)** (from OpenAI) released with the paper [Language Models are Unsupervised Multitask Learners](https://blog.openai.com/better-language-models/) by Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.
-4. **[Transformer-XL](https://github.com/kimiyoung/transformer-xl)** (from Google/CMU) released with the paper [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860) by Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.
-5. **[XLNet](https://github.com/zihangdai/xlnet/)** (from Google/CMU) released with the paper [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.
-6. **[XLM](https://github.com/facebookresearch/XLM/)** (from Facebook) released together with the paper [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample and Alexis Conneau.
-7. **[RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/roberta)** (from Facebook), released together with the paper a [Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
-8. **[DistilBERT](https://github.com/huggingface/transformers/tree/master/examples/distillation)** (from HuggingFace), released together with the paper [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108) by Victor Sanh, Lysandre Debut and Thomas Wolf. The same method has been applied to compress GPT2 into [DistilGPT2](https://github.com/huggingface/transformers/tree/master/examples/distillation), RoBERTa into [DistilRoBERTa](https://github.com/huggingface/transformers/tree/master/examples/distillation), Multilingual BERT into [DistilmBERT](https://github.com/huggingface/transformers/tree/master/examples/distillation) and a German version of DistilBERT.
-9. **[CTRL](https://github.com/salesforce/ctrl/)** (from Salesforce) released with the paper [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://arxiv.org/abs/1909.05858) by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and Richard Socher.
-10. **[CamemBERT](https://camembert-model.fr)** (from Inria/Facebook/Sorbonne) released with the paper [CamemBERT: a Tasty French Language Model](https://arxiv.org/abs/1911.03894) by Louis Martin*, Benjamin Muller*, Pedro Javier Ortiz Suárez*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.
-11. **[ALBERT](https://github.com/google-research/ALBERT)** (from Google Research and the Toyota Technological Institute at Chicago) released with the paper [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942), by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.
-12. **[T5](https://github.com/google-research/text-to-text-transfer-transformer)** (from Google AI) released with the paper [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683) by Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.
-13. **[XLM-RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/xlmr)** (from Facebook AI), released together with the paper [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.
+1. **[BERT](https://huggingface.co/transformers/model_doc/bert.html)** (from Google) released with the paper [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.
+2. **[GPT](https://huggingface.co/transformers/model_doc/gpt.html)** (from OpenAI) released with the paper [Improving Language Understanding by Generative Pre-Training](https://blog.openai.com/language-unsupervised/) by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.
+3. **[GPT-2](https://huggingface.co/transformers/model_doc/gpt2.html)** (from OpenAI) released with the paper [Language Models are Unsupervised Multitask Learners](https://blog.openai.com/better-language-models/) by Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.
+4. **[Transformer-XL](https://huggingface.co/transformers/model_doc/transformerxl.html)** (from Google/CMU) released with the paper [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860) by Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.
+5. **[XLNet](https://huggingface.co/transformers/model_doc/xlnet.html)** (from Google/CMU) released with the paper [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.
+6. **[XLM](https://huggingface.co/transformers/model_doc/xlm.html)** (from Facebook) released together with the paper [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample and Alexis Conneau.
+7. **[RoBERTa](https://huggingface.co/transformers/model_doc/roberta.html)** (from Facebook), released together with the paper a [Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
+8. **[DistilBERT](https://huggingface.co/transformers/model_doc/distilbert.html)** (from HuggingFace), released together with the paper [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108) by Victor Sanh, Lysandre Debut and Thomas Wolf. The same method has been applied to compress GPT2 into [DistilGPT2](https://github.com/huggingface/transformers/tree/master/examples/distillation), RoBERTa into [DistilRoBERTa](https://github.com/huggingface/transformers/tree/master/examples/distillation), Multilingual BERT into [DistilmBERT](https://github.com/huggingface/transformers/tree/master/examples/distillation) and a German version of DistilBERT.
+9. **[CTRL](https://huggingface.co/transformers/model_doc/ctrl.html)** (from Salesforce) released with the paper [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://arxiv.org/abs/1909.05858) by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and Richard Socher.
+10. **[CamemBERT](https://huggingface.co/transformers/model_doc/camembert.html)** (from Inria/Facebook/Sorbonne) released with the paper [CamemBERT: a Tasty French Language Model](https://arxiv.org/abs/1911.03894) by Louis Martin*, Benjamin Muller*, Pedro Javier Ortiz Suárez*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.
+11. **[ALBERT](https://huggingface.co/transformers/model_doc/albert.html)** (from Google Research and the Toyota Technological Institute at Chicago) released with the paper [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942), by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.
+12. **[T5](https://huggingface.co/transformers/model_doc/t5.html)** (from Google AI) released with the paper [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683) by Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.
+13. **[XLM-RoBERTa](https://huggingface.co/transformers/model_doc/xlmroberta.html)** (from Facebook AI), released together with the paper [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.
 14. **[MMBT](https://github.com/facebookresearch/mmbt/)** (from Facebook), released together with the paper a [Supervised Multimodal Bitransformers for Classifying Images and Text](https://arxiv.org/pdf/1909.02950.pdf) by Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine.
-15. **[FlauBERT](https://github.com/getalp/Flaubert)** (from CNRS) released with the paper [FlauBERT: Unsupervised Language Model Pre-training for French](https://arxiv.org/abs/1912.05372) by Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, Didier Schwab.
-16. **[BART](https://github.com/pytorch/fairseq/tree/master/examples/bart)** (from Facebook) released with the paper [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://arxiv.org/pdf/1910.13461.pdf) by Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer.
-17. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users).
-18. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.
+15. **[FlauBERT](https://huggingface.co/transformers/model_doc/flaubert.html)** (from CNRS) released with the paper [FlauBERT: Unsupervised Language Model Pre-training for French](https://arxiv.org/abs/1912.05372) by Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, Didier Schwab.
+16. **[BART](https://huggingface.co/transformers/model_doc/bart.html)** (from Facebook) released with the paper [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://arxiv.org/pdf/1910.13461.pdf) by Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer.
+17. **[ELECTRA](https://huggingface.co/transformers/model_doc/electra.html)** (from Google Research/Stanford University) released with the paper [ELECTRA: Pre-training text encoders as discriminators rather than generators](https://arxiv.org/abs/2003.10555) by Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.
+18. **[DialoGPT](https://huggingface.co/transformers/model_doc/dialogpt.html)** (from Microsoft Research) released with the paper [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://arxiv.org/abs/1911.00536) by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, Bill Dolan.
+18. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users).
+19. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.

 These implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations (e.g. ~93 F1 on SQuAD for BERT Whole-Word-Masking, ~88 F1 on RocStories for OpenAI GPT, ~18.3 perplexity on WikiText 103 for Transformer-XL, ~0.916 Peason R coefficient on STS-B for XLNet). You can find more details on the performances in the Examples section of the [documentation](https://huggingface.co/transformers/examples.html).

@@ -336,7 +338,6 @@ python ./examples/run_glue.py \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
-    --do_lower_case \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=8   \
@@ -390,7 +391,6 @@ python -m torch.distributed.launch --nproc_per_node 8 ./examples/run_glue.py   \
    --task_name MRPC \
    --do_train   \
    --do_eval   \
-    --do_lower_case   \
    --data_dir $GLUE_DIR/MRPC/   \
    --max_seq_length 128   \
    --per_gpu_eval_batch_size=8   \
@@ -423,7 +423,6 @@ python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
    --do_eval \
-    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
@@ -537,6 +536,8 @@ You can create `Pipeline` objects for the following down-stream tasks:
 - `text-classification`: Initialize a `TextClassificationPipeline` directly, or see `sentiment-analysis` for an example.
 - `question-answering`: Provided some context and a question refering to the context, it will extract the answer to the question in the context.
 - `fill-mask`: Takes an input sequence containing a masked token (e.g. `<mask>`) and return list of most probable filled sequences, with their probabilities.
+ - `summarization`
+ - `translation_xx_to_yy`

 ```python
 from transformers import pipeline
@@ -26,7 +26,7 @@ author = u'huggingface'
 # The short X.Y version
 version = u''
 # The full version, including alpha/beta/rc tags
-release = u'2.7.0'
+release = u'2.8.0'


 # -- General configuration ---------------------------------------------------
@@ -104,3 +104,5 @@ The library currently contains PyTorch and Tensorflow implementations, pre-train
    model_doc/flaubert
    model_doc/bart
    model_doc/t5
+    model_doc/electra
+    model_doc/dialogpt
@@ -1,16 +1,38 @@
 Tokenizer
 ----------------------------------------------------

-The base class ``PreTrainedTokenizer`` implements the common methods for loading/saving a tokenizer either from a local file or directory, or from a pretrained tokenizer provided by the library (downloaded from HuggingFace's AWS S3 repository).
+A tokenizer is in charge of preparing the inputs for a model. The library comprise tokenizers for all the models. Most of the tokenizers are available in two flavors: a full python implementation and a "Fast" implementation based on the Rust library `tokenizers`. The "Fast" implementations allows (1) a significant speed-up in particular when doing batched tokenization and (2) additional methods to map between the original string (character and words) and the token space (e.g. getting the index of the token comprising a given character or the span of characters corresponding to a given token). Currently no "Fast" implementation is available for the SentencePiece-based tokenizers (for T5, ALBERT, CamemBERT, XLMRoBERTa and XLNet models).

-``PreTrainedTokenizer`` is the main entry point into tokenizers as it also implements the main methods for using all the tokenizers:
+The base classes ``PreTrainedTokenizer`` and ``PreTrainedTokenizerFast`` implements the common methods for encoding string inputs in model inputs (see below) and instantiating/saving python and "Fast" tokenizers either from a local file or directory or from a pretrained tokenizer provided by the library (downloaded from HuggingFace's AWS S3 repository).

- tokenizing, converting tokens to ids and back and encoding/decoding,
+``PreTrainedTokenizer`` and ``PreTrainedTokenizerFast`` thus implements the main methods for using all the tokenizers:
+
+- tokenizing (spliting strings in sub-word token strings), converting tokens strings to ids and back, and encoding/decoding (i.e. tokenizing + convert to integers),
 - adding new tokens to the vocabulary in a way that is independant of the underlying structure (BPE, SentencePiece...),
- managing special tokens (adding them, assigning them to roles, making sure they are not split during tokenization)
+- managing special tokens like mask, beginning-of-sentence, etc tokens (adding them, assigning them to attributes in the tokenizer for easy access and making sure they are not split during tokenization)
+
+``BatchEncoding`` holds the output of the tokenizer's encoding methods (``encode_plus`` and ``batch_encode_plus``) and is derived from a Python dictionary. When the tokenizer is a pure python tokenizer, this class behave just like a standard python dictionary and hold the various model inputs computed by these methodes (``input_ids``, ``attention_mask``...). When the tokenizer is a "Fast" tokenizer (i.e. backed by HuggingFace tokenizers library), this class provides in addition several advanced alignement methods which can be used to map between the original string (character and words) and the token space (e.g. getting the index of the token comprising a given character or the span of characters corresponding to a given token).

 ``PreTrainedTokenizer``
 ~~~~~~~~~~~~~~~~~~~~~~~~

 .. autoclass:: transformers.PreTrainedTokenizer
    :members:
+
+``PreTrainedTokenizerFast``
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.PreTrainedTokenizerFast
+    :members:
+
+``BatchEncoding``
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.BatchEncoding
+    :members:
+
+``SpecialTokensMixin``
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.SpecialTokensMixin
+    :members:
@@ -27,7 +27,7 @@ loss = outputs[0]
 # In transformers you can also have access to the logits:
 loss, logits = outputs[:2]

-# And even the attention weigths if you configure the model to output them (and other outputs too, see the docstrings and documentation)
+# And even the attention weights if you configure the model to output them (and other outputs too, see the docstrings and documentation)
 model = BertForSequenceClassification.from_pretrained('bert-base-uncased', output_attentions=True)
 outputs = model(input_ids, labels=labels)
 loss, logits, attentions = outputs
@@ -30,6 +30,8 @@ Tips:
  similar to a BERT-like architecture with the same number of hidden layers as it has to iterate through the same
  number of (repeating) layers.

+The original code can be found `here <https://github.com/google-research/ALBERT>`_.
+
 AlbertConfig
 ~~~~~~~~~~~~~~~~~~~~~

@@ -35,6 +35,8 @@ Tips:
  prediction rather than a token prediction. However, averaging over the sequence may yield better results than using
  the [CLS] token.

+The original code can be found `here <https://github.com/google-research/bert>`_.
+
 BertConfig
 ~~~~~~~~~~~~~~~~~~~~~

@@ -50,6 +52,13 @@ BertTokenizer
        create_token_type_ids_from_sequences, save_vocabulary


+BertTokenizerFast
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.BertTokenizerFast
+    :members:
+
+
 BertModel
 ~~~~~~~~~~~~~~~~~~~~

@@ -22,6 +22,8 @@ Tips:
 - This implementation is the same as RoBERTa. Refer to the `documentation of RoBERTa <./roberta.html>`__ for usage
  examples as well as the information relative to the inputs and outputs.

+The original code can be found `here <https://camembert-model.fr/>`_.
+
 CamembertConfig
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

@@ -31,6 +31,8 @@ Tips:
  See `reusing the past in generative models <../quickstart.html#using-the-past>`_ for more information on the usage
  of this argument.

+The original code can be found `here <https://github.com/salesforce/ctrl>`_.
+

 CTRLConfig
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -0,0 +1,29 @@
+DialoGPT
+----------------------------------------------------
+
+Overview
+~~~~~~~~~~~~~~~~~~~~~
+
+DialoGPT was proposed in
+`DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation <https://arxiv.org/abs/1911.00536>`_
+by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, Bill Dolan.
+It's a GPT2 Model trained on 147M conversation-like exchanges extracted from Reddit.
+
+The abstract from the paper is the following:
+
+*We present a large, tunable neural conversational response generation model, DialoGPT (dialogue generative pre-trained transformer). 
+Trained on 147M conversation-like exchanges extracted from Reddit comment chains over a period spanning from 2005 through 2017, DialoGPT extends the Hugging Face PyTorch transformer to attain a performance close to human both in terms of automatic and human evaluation in single-turn dialogue settings.
+We show that conversational systems that leverage DialoGPT generate more relevant, contentful and context-consistent responses than strong baseline systems.
+The pre-trained model and training pipeline are publicly released to facilitate research into neural response generation and the development of more intelligent open-domain dialogue systems.*
+
+Tips:
+
+- DialoGPT is a model with absolute position embeddings so it's usually advised to pad the inputs on
+  the right rather than the left.
+- DialoGPT was trained with a causal language modeling (CLM) objective on conversational data and is therefore powerful at response generation in open-domain dialogue systems.
+- DialoGPT enables the user to create a chat bot in just 10 lines of code as shown on `DialoGPT's model card <https://huggingface.co/microsoft/DialoGPT-medium>`_.
+ 
+
+DialoGPT's architecture is based on the GPT2 model, so one can refer to GPT2's `docstring <https://huggingface.co/transformers/model_doc/gpt2.html>`_.
+
+The original code can be found `here <https://github.com/microsoft/DialoGPT>`_.
@@ -27,6 +27,8 @@ Tips:
 - DistilBert doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just separate your segments with the separation token `tokenizer.sep_token` (or `[SEP]`)
 - DistilBert doesn't have options to select the input positions (`position_ids` input). This could be added if necessary though, just let's us know if you need this option.

+The original code can be found `here <https://github.com/huggingface/transformers/tree/master/examples/distillation>`_.
+

 DistilBertConfig
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -42,6 +44,13 @@ DistilBertTokenizer
    :members:


+DistilBertTokenizerFast
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.DistilBertTokenizerFast
+    :members:
+
+
 DistilBertModel
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

@@ -0,0 +1,124 @@
+ELECTRA
+----------------------------------------------------
+
+The ELECTRA model was proposed in the paper.
+`ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators <https://openreview.net/pdf?id=r1xMH1BtvB>`__.
+ELECTRA is a new pre-training approach which trains two transformer models: the generator and the discriminator. The
+generator's role is to replace tokens in a sequence, and is therefore trained as a masked language model. The discriminator,
+which is the model we're interested in, tries to identify which tokens were replaced by the generator in the sequence.
+
+The abstract from the paper is the following:
+
+*Masked language modeling (MLM) pre-training methods such as BERT corrupt
+the input by replacing some tokens with [MASK] and then train a model to
+reconstruct the original tokens. While they produce good results when transferred
+to downstream NLP tasks, they generally require large amounts of compute to be
+effective. As an alternative, we propose a more sample-efficient pre-training task
+called replaced token detection. Instead of masking the input, our approach
+corrupts it by replacing some tokens with plausible alternatives sampled from a small
+generator network. Then, instead of training a model that predicts the original
+identities of the corrupted tokens, we train a discriminative model that predicts
+whether each token in the corrupted input was replaced by a generator sample
+or not. Thorough experiments demonstrate this new pre-training task is more
+efficient than MLM because the task is defined over all input tokens rather than
+just the small subset that was masked out. As a result, the contextual representations
+learned by our approach substantially outperform the ones learned by BERT
+given the same model size, data, and compute. The gains are particularly strong
+for small models; for example, we train a model on one GPU for 4 days that
+outperforms GPT (trained using 30x more compute) on the GLUE natural language
+understanding benchmark. Our approach also works well at scale, where it
+performs comparably to RoBERTa and XLNet while using less than 1/4 of their
+compute and outperforms them when using the same amount of compute.*
+
+Tips:
+
+- ELECTRA is the pre-training approach, therefore there is nearly no changes done to the underlying model: BERT. The
+  only change is the separation of the embedding size and the hidden size -> The embedding size is generally smaller,
+  while the hidden size is larger. An additional projection layer (linear) is used to project the embeddings from
+  their embedding size to the hidden size. In the case where the embedding size is the same as the hidden size, no
+  projection layer is used.
+- The ELECTRA checkpoints saved using `Google Research's implementation <https://github.com/google-research/electra>`__
+  contain both the generator and discriminator. The conversion script requires the user to name which model to export
+  into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all
+  available ELECTRA models, however. This means that the discriminator may be loaded in the `ElectraForMaskedLM` model,
+  and the generator may be loaded in the `ElectraForPreTraining` model (the classification head will be randomly
+  initialized as it doesn't exist in the generator).
+
+The original code can be found `here <https://github.com/google-research/electra>`_.
+
+
+ElectraConfig
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.ElectraConfig
+    :members:
+
+
+ElectraTokenizer
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.ElectraTokenizer
+    :members:
+
+
+ElectraTokenizerFast
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.ElectraTokenizerFast
+    :members:
+
+
+ElectraModel
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.ElectraModel
+    :members:
+
+
+ElectraForPreTraining
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.ElectraForPreTraining
+    :members:
+
+
+ElectraForMaskedLM
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.ElectraForMaskedLM
+    :members:
+
+
+ElectraForTokenClassification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.ElectraForTokenClassification
+    :members:
+
+
+TFElectraModel
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFElectraModel
+    :members:
+
+
+TFElectraForPreTraining
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFElectraForPreTraining
+    :members:
+
+
+TFElectraForMaskedLM
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFElectraForMaskedLM
+    :members:
+
+
+TFElectraForTokenClassification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFElectraForTokenClassification
+    :members:
@@ -20,6 +20,8 @@ of the time they outperform other pre-training approaches. Different versions of
 evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared
 to the research community for further reproducible experiments in French NLP.*

+The original code can be found `here <https://github.com/getalp/Flaubert>`_.
+

 FlaubertConfig
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -36,6 +36,9 @@ Tips:
 `Write With Transformer <https://transformer.huggingface.co/doc/gpt>`__ is a webapp created and hosted by
 Hugging Face showcasing the generative capabilities of several models. GPT is one of them.

+The original code can be found `here <https://github.com/openai/finetune-transformer-lm>`_.
+
+
 OpenAIGPTConfig
 ~~~~~~~~~~~~~~~~~~~~~

@@ -50,6 +53,13 @@ OpenAIGPTTokenizer
    :members: save_vocabulary


+OpenAIGPTTokenizerFast
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.OpenAIGPTTokenizerFast
+    :members:
+
+
 OpenAIGPTModel
 ~~~~~~~~~~~~~~~~~~~~~~~~~

@@ -34,6 +34,8 @@ Tips:
 Hugging Face showcasing the generative capabilities of several models. GPT-2 is one of them and is available in five
 different sizes: small, medium, large, xl and a distilled version of the small checkpoint: distilgpt-2.

+The original code can be found `here <https://openai.com/blog/better-language-models/>`_.
+

 GPT2Config
 ~~~~~~~~~~~~~~~~~~~~~
@@ -49,6 +51,13 @@ GPT2Tokenizer
    :members: save_vocabulary


+GPT2TokenizerFast
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.GPT2TokenizerFast
+    :members:
+
+
 GPT2Model
 ~~~~~~~~~~~~~~~~~~~~~

@@ -28,6 +28,9 @@ Tips:
 - RoBERTa doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just separate your segments with the separation token `tokenizer.sep_token` (or `</s>`)
 - `Camembert <./camembert.html>`__ is a wrapper around RoBERTa. Refer to this page for usage examples.

+The original code can be found `here <https://github.com/pytorch/fairseq/tree/master/examples/roberta>`_.
+
+
 RobertaConfig
 ~~~~~~~~~~~~~~~~~~~~~

@@ -43,6 +46,13 @@ RobertaTokenizer
        create_token_type_ids_from_sequences, save_vocabulary


+RobertaTokenizerFast
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.RobertaTokenizerFast
+    :members: build_inputs_with_special_tokens
+
+
 RobertaModel
 ~~~~~~~~~~~~~~~~~~~~

@@ -57,6 +57,8 @@ Tips
 - For sequence to sequence generation, it is recommended to use ``T5ForConditionalGeneration.generate()``. The method takes care of feeding the encoded input via cross-attention layers to the decoder and auto-regressively generates the decoder output.
 - T5 uses relative scalar embeddings. Encoder input padding can be done on the left and on the right.

+The original code can be found `here <https://github.com/google-research/text-to-text-transfer-transformer>`_.
+

 T5Config
 ~~~~~~~~~~~~~~~~~~~~~
@@ -30,6 +30,8 @@ Tips:
  The original implementation trains on SQuAD with padding on the left, therefore the padding defaults are set to left.
 - Transformer-XL is one of the few models that has no sequence length limit.

+The original code can be found `here <https://github.com/kimiyoung/transformer-xl>`_.
+

 TransfoXLConfig
 ~~~~~~~~~~~~~~~~~~~~~
@@ -45,6 +47,13 @@ TransfoXLTokenizer
    :members: save_vocabulary


+TransfoXLTokenizerFast
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TransfoXLTokenizerFast
+    :members:
+
+
 TransfoXLModel
 ~~~~~~~~~~~~~~~~~~~~~~~~~~

@@ -30,6 +30,8 @@ Tips:
 - XLM has multilingual checkpoints which leverage a specific `lang` parameter. Check out the
  `multi-lingual <../multilingual.html>`__ page for more information.

+The original code can be found `here <https://github.com/facebookresearch/XLM/>`_.
+

 XLMConfig
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -28,6 +28,9 @@ Tips:
 - This implementation is the same as RoBERTa. Refer to the `documentation of RoBERTa <./roberta.html>`__ for usage
  examples as well as the information relative to the inputs and outputs.

+The original code can be found `here <https://github.com/pytorch/fairseq/tree/master/examples/xlmr>`_.
+
+
 XLMRobertaConfig
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

@@ -32,6 +32,8 @@ Tips:
  `target_mapping` inputs to control the attention span and outputs (see examples in `examples/run_generation.py`)
 - XLNet is one of the few models that has no sequence length limit.

+The original code can be found `here <https://github.com/zihangdai/xlnet/>`_.
+

 XLNetConfig
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -0,0 +1 @@
+../../notebooks/README.md
@@ -1,16 +0,0 @@
-Notebooks
-================================================
-
-We include `three Jupyter Notebooks <https://github.com/huggingface/transformers/tree/master/notebooks>`_ that can be used to check that the predictions of the PyTorch model are identical to the predictions of the original TensorFlow model.
-
-
-*
-  The first NoteBook (\ `Comparing-TF-and-PT-models.ipynb <https://github.com/huggingface/transformers/blob/master/notebooks/Comparing-TF-and-PT-models.ipynb>`_\ ) extracts the hidden states of a full sequence on each layers of the TensorFlow and the PyTorch models and computes the standard deviation between them. In the given example, we get a standard deviation of 1.5e-7 to 9e-7 on the various hidden state of the models.
-
-*
-  The second NoteBook (\ `Comparing-TF-and-PT-models-SQuAD.ipynb <https://github.com/huggingface/transformers/blob/master/notebooks/Comparing-TF-and-PT-models-SQuAD.ipynb>`_\ ) compares the loss computed by the TensorFlow and the PyTorch models for identical initialization of the fine-tuning layer of the ``BertForQuestionAnswering`` and computes the standard deviation between them. In the given example, we get a standard deviation of 2.5e-7 between the models.
-
-*
-  The third NoteBook (\ `Comparing-TF-and-PT-models-MLM-NSP.ipynb <https://github.com/huggingface/transformers/blob/master/notebooks/Comparing-TF-and-PT-models-MLM-NSP.ipynb>`_\ ) compares the predictions computed by the TensorFlow and the PyTorch models for masked token language modeling using the pre-trained masked language modeling model.
-
-Please follow the instructions given in the notebooks to run and modify them.
@@ -283,4 +283,16 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
 |                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
 |                   | ``bart-large-cnn``                                         | | 12-layer, 1024-hidden, 16-heads, 406M parameters       (same as base)                                                               |
 |                   |                                                            | | bart-large base architecture finetuned on cnn summarization task                                                                    |
+|                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+|                   | ``mbart-large-en-ro``                                      | | 12-layer, 1024-hidden, 16-heads, 880M parameters                                                                                    |
+|                   |                                                            | | bart-large architecture pretrained on cc25 multilingual data , finetuned on WMT english romanian translation.                       |
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+| DialoGPT          | ``DialoGPT-small``                                         | | 12-layer, 768-hidden, 12-heads, 124M parameters                                                                                     |
+|                   |                                                            | | Trained on English text: 147M conversation-like exchanges extracted from Reddit.                                                    |
+|                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+|                   | ``DialoGPT-medium``                                        | | 24-layer, 1024-hidden, 16-heads, 355M parameters                                                                                    |
+|                   |                                                            | | Trained on English text: 147M conversation-like exchanges extracted from Reddit.                                                    |
+|                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+|                   | ``DialoGPT-large``                                         | | 36-layer, 1280-hidden, 20-heads, 774M parameters                                                                                    |
+|                   |                                                            | | Trained on English text: 147M conversation-like exchanges extracted from Reddit.                                                    |
 +-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
@@ -58,14 +58,14 @@ where

 ``Uncased`` means that the text has been lowercased before WordPiece tokenization, e.g., ``John Smith`` becomes ``john smith``. The Uncased model also strips out any accent markers. ``Cased`` means that the true case and accent markers are preserved. Typically, the Uncased model is better unless you know that case information is important for your task (e.g., Named Entity Recognition or Part-of-Speech tagging). For information about the Multilingual and Chinese model, see the `Multilingual README <https://github.com/google-research/bert/blob/master/multilingual.md>`__ or the original TensorFlow repository.

-When using an ``uncased model``\ , make sure to pass ``--do_lower_case`` to the example training scripts (or pass ``do_lower_case=True`` to FullTokenizer if you're using your own script and loading the tokenizer your-self.).
+When using an ``uncased model``\ , make sure your tokenizer has ``do_lower_case=True`` (either in its configuration, or passed as an additional parameter).

 Examples:

 .. code-block:: python

   # BERT
-   tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True, do_basic_tokenize=True)
+   tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_basic_tokenize=True)
   model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

   # OpenAI GPT
@@ -140,13 +140,13 @@ Here is the recommended way of saving the model, configuration and vocabulary to

   torch.save(model_to_save.state_dict(), output_model_file)
   model_to_save.config.to_json_file(output_config_file)
-   tokenizer.save_vocabulary(output_dir)
+   tokenizer.save_pretrained(output_dir)

   # Step 2: Re-load the saved model and vocabulary

   # Example for a Bert model
   model = BertForQuestionAnswering.from_pretrained(output_dir)
-   tokenizer = BertTokenizer.from_pretrained(output_dir, do_lower_case=args.do_lower_case)  # Add specific options if needed
+   tokenizer = BertTokenizer.from_pretrained(output_dir)  # Add specific options if needed
   # Example for a GPT model
   model = OpenAIGPTDoubleHeadsModel.from_pretrained(output_dir)
   tokenizer = OpenAIGPTTokenizer.from_pretrained(output_dir)
@@ -17,6 +17,7 @@ pip install -r ./examples/requirements.txt
 | Section                    | Description                                                                                                                                                |
 |----------------------------|------------------------------------------------------------------------------------------------------------------------------------------
 | [TensorFlow 2.0 models on GLUE](#TensorFlow-2.0-Bert-models-on-GLUE) | Examples running BERT TensorFlow 2.0 model on the GLUE tasks. |
+| [Running on TPUs](#running-on-tpus) | Examples on running fine-tuning tasks on Google TPUs to accelerate workloads. |
 | [Language Model training](#language-model-training) | Fine-tuning (or training from scratch) the library models for language modeling on a text dataset. Causal language modeling for GPT/GPT-2, masked language modeling for BERT/RoBERTa. |
 | [Language Generation](#language-generation) | Conditional text generation using the auto-regressive models of the library: GPT, GPT-2, Transformer-XL and XLNet. |
 | [GLUE](#glue) | Examples running BERT/XLM/XLNet/RoBERTa on the 9 GLUE tasks. Examples feature distributed training as well as half-precision. |
@@ -48,12 +49,54 @@ Quick benchmarks from the script (no other modifications):

 Mixed precision (AMP) reduces the training time considerably for the same hardware and hyper-parameters (same batch size was used).

+## Running on TPUs
+
+You can accelerate your workloads on Google's TPUs. For information on how to setup your TPU environment refer to this
+[README](https://github.com/pytorch/xla/blob/master/README.md).
+
+The following are some examples of running the `*_tpu.py` finetuning scripts on TPUs. All steps for data preparation are
+identical to your normal GPU + Huggingface setup.
+
+### GLUE
+
+Before running anyone of these GLUE tasks you should download the
+[GLUE data](https://gluebenchmark.com/tasks) by running
+[this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e)
+and unpack it to some directory `$GLUE_DIR`.
+
+For running your GLUE task on MNLI dataset you can run something like the following:
+
+```
+export XRT_TPU_CONFIG="tpu_worker;0;$TPU_IP_ADDRESS:8470"
+export GLUE_DIR=/path/to/glue
+export TASK_NAME=MNLI
+
+python run_glue_tpu.py \
+  --model_type bert \
+  --model_name_or_path bert-base-cased \
+  --task_name $TASK_NAME \
+  --do_train \
+  --do_eval \
+  --data_dir $GLUE_DIR/$TASK_NAME \
+  --max_seq_length 128 \
+  --train_batch_size 32 \
+  --learning_rate 3e-5 \
+  --num_train_epochs 3.0 \
+  --output_dir /tmp/$TASK_NAME \
+  --overwrite_output_dir \
+  --logging_steps 50 \
+  --save_steps 200 \
+  --num_cores=8 \
+  --only_log_master
+```
+
+
 ## Language model training

 Based on the script [`run_language_modeling.py`](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py).

-Fine-tuning (or training from scratch) the library models for language modeling on a text dataset for GPT, GPT-2, BERT and RoBERTa (DistilBERT 
-to be added soon). GPT and GPT-2 are fine-tuned using a causal language modeling (CLM) loss while BERT and RoBERTa 
+Fine-tuning (or training from scratch) the library models for language modeling on a text dataset for GPT, GPT-2, BERT and RoBERTa (DistilBERT
+to be added soon). GPT and GPT-2 are fine-tuned using a causal language modeling (CLM) loss while BERT and RoBERTa
 are fine-tuned using a masked language modeling (MLM) loss.

 Before running the following example, you should get a file that contains text on which the language model will be
@@ -168,7 +211,6 @@ python run_glue.py \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
-  --do_lower_case \
  --data_dir $GLUE_DIR/$TASK_NAME \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 32 \
@@ -209,7 +251,6 @@ python run_glue.py \
  --task_name MRPC \
  --do_train \
  --do_eval \
-  --do_lower_case \
  --data_dir $GLUE_DIR/MRPC/ \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 32 \
@@ -236,7 +277,6 @@ python run_glue.py \
  --task_name MRPC \
  --do_train \
  --do_eval \
-  --do_lower_case \
  --data_dir $GLUE_DIR/MRPC/ \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 32 \
@@ -261,7 +301,6 @@ python -m torch.distributed.launch \
    --task_name MRPC \
    --do_train \
    --do_eval \
-    --do_lower_case \
    --data_dir $GLUE_DIR/MRPC/ \
    --max_seq_length 128 \
    --per_gpu_train_batch_size 8 \
@@ -295,7 +334,6 @@ python -m torch.distributed.launch \
    --task_name mnli \
    --do_train \
    --do_eval \
-    --do_lower_case \
    --data_dir $GLUE_DIR/MNLI/ \
    --max_seq_length 128 \
    --per_gpu_train_batch_size 8 \
@@ -336,7 +374,6 @@ python ./examples/run_multiple_choice.py \
 --model_name_or_path roberta-base \
 --do_train \
 --do_eval \
--do_lower_case \
 --data_dir $SWAG_DIR \
 --learning_rate 5e-5 \
 --num_train_epochs 3 \
@@ -382,7 +419,6 @@ python run_squad.py \
  --model_name_or_path bert-base-uncased \
  --do_train \
  --do_eval \
-  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
@@ -411,7 +447,6 @@ python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
    --do_eval \
-    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
@@ -447,7 +482,6 @@ python run_squad.py \
    --model_name_or_path xlnet-large-cased \
    --do_train \
    --do_eval \
-    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
@@ -597,7 +631,6 @@ python examples/hans/test_hans.py \
        --task_name hans \
        --model_type $MODEL_TYPE \
        --do_eval \
-        --do_lower_case \
        --data_dir $HANS_DIR \
        --model_name_or_path $MODEL_PATH \
        --max_seq_length 128 \
@@ -20,9 +20,10 @@

 import argparse
 import csv
+import logging
 import timeit
 from time import time
-from typing import List
+from typing import Callable, List

 from transformers import (
    AutoConfig,
@@ -46,10 +47,8 @@ if is_torch_available():

 input_text = """Bent over their instruments, three hundred Fertilizers were plunged, as
 the Director of Hatcheries and Conditioning entered the room, in the
-
-
-
 scarcely breathing silence, the absent-minded, soliloquizing hum or
+
 whistle, of absorbed concentration. A troop of newly arrived students,
 very young, pink and callow, followed nervously, rather abjectly, at the
 Director's heels. Each of them carried a notebook, in which, whenever
@@ -271,8 +270,9 @@ def create_setup_and_compute(
    amp: bool = False,
    fp16: bool = False,
    save_to_csv: bool = False,
-    csv_filename: str = f"results_{round(time())}.csv",
+    csv_time_filename: str = f"time_{round(time())}.csv",
    csv_memory_filename: str = f"memory_{round(time())}.csv",
+    print_fn: Callable[[str], None] = print,
 ):
    if xla:
        tf.config.optimizer.set_jit(True)
@@ -282,7 +282,16 @@ def create_setup_and_compute(
    if tensorflow:
        dictionary = {model_name: {} for model_name in model_names}
        results = _compute_tensorflow(
-            model_names, batch_sizes, slice_sizes, dictionary, average_over, amp, no_speed, no_memory, verbose
+            model_names,
+            batch_sizes,
+            slice_sizes,
+            dictionary,
+            average_over,
+            amp,
+            no_speed,
+            no_memory,
+            verbose,
+            print_fn,
        )
    else:
        device = "cuda" if (gpu and torch.cuda.is_available()) else "cpu"
@@ -299,100 +308,107 @@ def create_setup_and_compute(
            no_speed,
            no_memory,
            verbose,
+            print_fn,
        )

-    print("=========== RESULTS ===========")
+    print_fn("=========== RESULTS ===========")
    for model_name in model_names:
-        print("\t" + f"======= MODEL CHECKPOINT: {model_name} =======")
+        print_fn("\t" + f"======= MODEL CHECKPOINT: {model_name} =======")
        for batch_size in results[model_name]["bs"]:
-            print("\t\t" + f"===== BATCH SIZE: {batch_size} =====")
+            print_fn("\t\t" + f"===== BATCH SIZE: {batch_size} =====")
            for slice_size in results[model_name]["ss"]:
-                result = results[model_name]["results"][batch_size][slice_size]
+                time = results[model_name]["time"][batch_size][slice_size]
                memory = results[model_name]["memory"][batch_size][slice_size]
-                if isinstance(result, str):
-                    print(f"\t\t{model_name}/{batch_size}/{slice_size}: " f"{result} " f"{memory}")
+                if isinstance(time, str):
+                    print_fn(f"\t\t{model_name}/{batch_size}/{slice_size}: " f"{time} " f"{memory}")
                else:
-                    print(
+                    print_fn(
                        f"\t\t{model_name}/{batch_size}/{slice_size}: "
-                        f"{(round(1000 * result) / 1000)}"
+                        f"{(round(1000 * time) / 1000)}"
                        f"s "
                        f"{memory}"
                    )

    if save_to_csv:
-        with open(csv_filename, mode="w") as csv_file, open(csv_memory_filename, mode="w") as csv_memory_file:
-            fieldnames = [
-                "model",
-                "1x8",
-                "1x64",
-                "1x128",
-                "1x256",
-                "1x512",
-                "1x1024",
-                "2x8",
-                "2x64",
-                "2x128",
-                "2x256",
-                "2x512",
-                "2x1024",
-                "4x8",
-                "4x64",
-                "4x128",
-                "4x256",
-                "4x512",
-                "4x1024",
-                "8x8",
-                "8x64",
-                "8x128",
-                "8x256",
-                "8x512",
-                "8x1024",
-            ]
+        with open(csv_time_filename, mode="w") as csv_time_file, open(
+            csv_memory_filename, mode="w"
+        ) as csv_memory_file:

-            writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
-            writer.writeheader()
-            memory_writer = csv.DictWriter(csv_memory_file, fieldnames=fieldnames)
+            assert len(model_names) > 0, "At least 1 model should be defined, but got {}".format(model_names)
+
+            fieldnames = ["model", "batch_size", "sequence_length"]
+            time_writer = csv.DictWriter(csv_time_file, fieldnames=fieldnames + ["time_in_s"])
+            time_writer.writeheader()
+            memory_writer = csv.DictWriter(csv_memory_file, fieldnames=fieldnames + ["memory"])
            memory_writer.writeheader()

            for model_name in model_names:
-                model_results = {
-                    f"{bs}x{ss}": results[model_name]["results"][bs][ss]
-                    for bs in results[model_name]["results"]
-                    for ss in results[model_name]["results"][bs]
-                }
-                writer.writerow({"model": model_name, **model_results})
+                time_dict = results[model_name]["time"]
+                memory_dict = results[model_name]["memory"]
+                for bs in time_dict:
+                    for ss in time_dict[bs]:
+                        time_writer.writerow(
+                            {
+                                "model": model_name,
+                                "batch_size": bs,
+                                "sequence_length": ss,
+                                "time_in_s": "{:.4f}".format(time_dict[bs][ss]),
+                            }
+                        )

-                model_memory_results = {
-                    f"{bs}x{ss}": results[model_name]["memory"][bs][ss]
-                    for bs in results[model_name]["memory"]
-                    for ss in results[model_name]["memory"][bs]
-                }
-                memory_writer.writerow({"model": model_name, **model_memory_results})
+                for bs in memory_dict:
+                    for ss in time_dict[bs]:
+                        memory_writer.writerow(
+                            {
+                                "model": model_name,
+                                "batch_size": bs,
+                                "sequence_length": ss,
+                                "memory": memory_dict[bs][ss],
+                            }
+                        )


-def print_summary_statistics(summary: MemorySummary):
-    print(
+def print_summary_statistics(summary: MemorySummary, print_fn: Callable[[str], None]):
+    print_fn(
        "\nLines by line memory consumption:\n"
        + "\n".join(
            f"{state.frame.filename}:{state.frame.line_number}: mem {state.cpu_gpu}: {state.frame.line_text}"
            for state in summary.sequential
        )
    )
-    print(
+    print_fn(
        "\nLines with top memory consumption:\n"
        + "\n".join(
            f"=> {state.frame.filename}:{state.frame.line_number}: mem {state.cpu_gpu}: {state.frame.line_text}"
            for state in summary.cumulative[:6]
        )
    )
-    print(
+    print_fn(
        "\nLines with lowest memory consumption:\n"
        + "\n".join(
            f"=> {state.frame.filename}:{state.frame.line_number}: mem {state.cpu_gpu}: {state.frame.line_text}"
            for state in summary.cumulative[-6:]
        )
    )
-    print(f"\nTotal memory increase: {summary.total}")
+    print_fn(f"\nTotal memory increase: {summary.total}")
+
+
+def get_print_function(save_print_log, log_filename):
+    if save_print_log:
+        logging.basicConfig(
+            level=logging.DEBUG,
+            filename=log_filename,
+            filemode="a+",
+            format="%(asctime)-15s %(levelname)-8s %(message)s",
+        )
+
+        def print_with_print_log(*args):
+            logging.info(*args)
+            print(*args)
+
+        return print_with_print_log
+    else:
+        return print


 def _compute_pytorch(
@@ -407,9 +423,10 @@ def _compute_pytorch(
    no_speed,
    no_memory,
    verbose,
+    print_fn,
 ):
    for c, model_name in enumerate(model_names):
-        print(f"{c + 1} / {len(model_names)}")
+        print_fn(f"{c + 1} / {len(model_names)}")
        config = AutoConfig.from_pretrained(model_name, torchscript=torchscript)
        model = AutoModel.from_pretrained(model_name, config=config)
        tokenizer = AutoTokenizer.from_pretrained(model_name)
@@ -418,10 +435,13 @@ def _compute_pytorch(

        max_input_size = tokenizer.max_model_input_sizes[model_name]

-        dictionary[model_name] = {"bs": batch_sizes, "ss": slice_sizes, "results": {}, "memory": {}}
-        dictionary[model_name]["results"] = {i: {} for i in batch_sizes}
+        dictionary[model_name] = {"bs": batch_sizes, "ss": slice_sizes, "time": {}, "memory": {}}
+        dictionary[model_name]["time"] = {i: {} for i in batch_sizes}
        dictionary[model_name]["memory"] = {i: {} for i in batch_sizes}

+        print_fn("Using model {}".format(model))
+        print_fn("Number of all parameters {}".format(model.num_parameters()))
+
        for batch_size in batch_sizes:
            if fp16:
                model.half()
@@ -430,12 +450,12 @@ def _compute_pytorch(

            for slice_size in slice_sizes:
                if max_input_size is not None and slice_size > max_input_size:
-                    dictionary[model_name]["results"][batch_size][slice_size] = "N/A"
+                    dictionary[model_name]["time"][batch_size][slice_size] = "N/A"
                else:
                    sequence = torch.tensor(tokenized_sequence[:slice_size], device=device).repeat(batch_size, 1)
                    try:
                        if torchscript:
-                            print("Tracing model with sequence size", sequence.shape)
+                            print_fn("Tracing model with sequence size {}".format(sequence.shape))
                            inference = torch.jit.trace(model, sequence)
                            inference(sequence)
                        else:
@@ -451,33 +471,33 @@ def _compute_pytorch(
                            summary = stop_memory_tracing(trace)

                            if verbose:
-                                print_summary_statistics(summary)
+                                print_summary_statistics(summary, print_fn)

                            dictionary[model_name]["memory"][batch_size][slice_size] = str(summary.total)
                        else:
                            dictionary[model_name]["memory"][batch_size][slice_size] = "N/A"

                        if not no_speed:
-                            print("Going through model with sequence of shape", sequence.shape)
+                            print_fn("Going through model with sequence of shape".format(sequence.shape))
                            runtimes = timeit.repeat(lambda: inference(sequence), repeat=average_over, number=3)
                            average_time = sum(runtimes) / float(len(runtimes)) / 3.0
-                            dictionary[model_name]["results"][batch_size][slice_size] = average_time
+                            dictionary[model_name]["time"][batch_size][slice_size] = average_time
                        else:
-                            dictionary[model_name]["results"][batch_size][slice_size] = "N/A"
+                            dictionary[model_name]["time"][batch_size][slice_size] = "N/A"

                    except RuntimeError as e:
-                        print("Doesn't fit on GPU.", e)
+                        print_fn("Doesn't fit on GPU. {}".format(e))
                        torch.cuda.empty_cache()
-                        dictionary[model_name]["results"][batch_size][slice_size] = "N/A"
+                        dictionary[model_name]["time"][batch_size][slice_size] = "N/A"
                        dictionary[model_name]["memory"][batch_size][slice_size] = "N/A"
    return dictionary


 def _compute_tensorflow(
-    model_names, batch_sizes, slice_sizes, dictionary, average_over, amp, no_speed, no_memory, verbose
+    model_names, batch_sizes, slice_sizes, dictionary, average_over, amp, no_speed, no_memory, verbose, print_fn
 ):
    for c, model_name in enumerate(model_names):
-        print(f"{c + 1} / {len(model_names)}")
+        print_fn(f"{c + 1} / {len(model_names)}")
        config = AutoConfig.from_pretrained(model_name)
        model = TFAutoModel.from_pretrained(model_name, config=config)
        tokenizer = AutoTokenizer.from_pretrained(model_name)
@@ -486,11 +506,12 @@ def _compute_tensorflow(

        max_input_size = tokenizer.max_model_input_sizes[model_name]

-        dictionary[model_name] = {"bs": batch_sizes, "ss": slice_sizes, "results": {}, "memory": {}}
-        dictionary[model_name]["results"] = {i: {} for i in batch_sizes}
+        dictionary[model_name] = {"bs": batch_sizes, "ss": slice_sizes, "time": {}, "memory": {}}
+        dictionary[model_name]["time"] = {i: {} for i in batch_sizes}
        dictionary[model_name]["memory"] = {i: {} for i in batch_sizes}

-        print("Using model", model)
+        print_fn("Using model {}".format(model))
+        print_fn("Number of all parameters {}".format(model.num_parameters()))

        @tf.function
        def inference(inputs):
@@ -499,14 +520,14 @@ def _compute_tensorflow(
        for batch_size in batch_sizes:
            for slice_size in slice_sizes:
                if max_input_size is not None and slice_size > max_input_size:
-                    dictionary[model_name]["results"][batch_size][slice_size] = "N/A"
+                    dictionary[model_name]["time"][batch_size][slice_size] = "N/A"
                else:
                    sequence = tf.stack(
                        [tf.squeeze(tf.constant(tokenized_sequence[:slice_size])[None, :])] * batch_size
                    )

                    try:
-                        print("Going through model with sequence of shape", sequence.shape)
+                        print_fn("Going through model with sequence of shape {}".format(sequence.shape))
                        # To make sure that the model is traced + that the tensors are on the appropriate device
                        inference(sequence)

@@ -517,7 +538,7 @@ def _compute_tensorflow(
                            summary = stop_memory_tracing(trace)

                            if verbose:
-                                print_summary_statistics(summary)
+                                print_summary_statistics(summary, print_fn)

                            dictionary[model_name]["memory"][batch_size][slice_size] = str(summary.total)
                        else:
@@ -526,13 +547,13 @@ def _compute_tensorflow(
                        if not no_speed:
                            runtimes = timeit.repeat(lambda: inference(sequence), repeat=average_over, number=3)
                            average_time = sum(runtimes) / float(len(runtimes)) / 3.0
-                            dictionary[model_name]["results"][batch_size][slice_size] = average_time
+                            dictionary[model_name]["time"][batch_size][slice_size] = average_time
                        else:
-                            dictionary[model_name]["results"][batch_size][slice_size] = "N/A"
+                            dictionary[model_name]["time"][batch_size][slice_size] = "N/A"

                    except tf.errors.ResourceExhaustedError as e:
-                        print("Doesn't fit on GPU.", e)
-                        dictionary[model_name]["results"][batch_size][slice_size] = "N/A"
+                        print_fn("Doesn't fit on GPU. {}".format(e))
+                        dictionary[model_name]["time"][batch_size][slice_size] = "N/A"
                        dictionary[model_name]["memory"][batch_size][slice_size] = "N/A"
    return dictionary

@@ -593,7 +614,25 @@ def main():
    )
    parser.add_argument("--save_to_csv", required=False, action="store_true", help="Save to a CSV file.")
    parser.add_argument(
-        "--csv_filename", required=False, default=None, help="CSV filename used if saving results to csv."
+        "--log_print", required=False, action="store_true", help="Save all print statements in log file."
+    )
+    parser.add_argument(
+        "--csv_time_filename",
+        required=False,
+        default=f"time_{round(time())}.csv",
+        help="CSV filename used if saving time results to csv.",
+    )
+    parser.add_argument(
+        "--csv_memory_filename",
+        required=False,
+        default=f"memory_{round(time())}.csv",
+        help="CSV filename used if saving memory results to csv.",
+    )
+    parser.add_argument(
+        "--log_filename",
+        required=False,
+        default=f"log_{round(time())}.txt",
+        help="Log filename used if print statements are saved in log.",
    )
    parser.add_argument(
        "--average_over", required=False, default=30, type=int, help="Times an experiment will be run."
@@ -614,11 +653,14 @@ def main():
            "distilgpt2",
            "roberta-base",
            "ctrl",
+            "t5-base",
+            "bart-large",
        ]
    else:
        args.models = args.models.split()

-    print("Running with arguments", args)
+    print_fn = get_print_function(args.log_print, args.log_filename)
+    print_fn("Running with arguments: {}".format(args))

    if args.torch:
        if is_torch_available():
@@ -631,11 +673,13 @@ def main():
                torchscript=args.torchscript,
                fp16=args.fp16,
                save_to_csv=args.save_to_csv,
-                csv_filename=args.csv_filename,
+                csv_time_filename=args.csv_time_filename,
+                csv_memory_filename=args.csv_memory_filename,
                average_over=args.average_over,
                no_speed=args.no_speed,
                no_memory=args.no_memory,
                verbose=args.verbose,
+                print_fn=print_fn,
            )
        else:
            raise ImportError("Trying to run a PyTorch benchmark but PyTorch was not found in the environment.")
@@ -650,11 +694,13 @@ def main():
                xla=args.xla,
                amp=args.amp,
                save_to_csv=args.save_to_csv,
-                csv_filename=args.csv_filename,
+                csv_time_filename=args.csv_time_filename,
+                csv_memory_filename=args.csv_memory_filename,
                average_over=args.average_over,
                no_speed=args.no_speed,
                no_memory=args.no_memory,
                verbose=args.verbose,
+                print_fn=print_fn,
            )
        else:
            raise ImportError("Trying to run a TensorFlow benchmark but TensorFlow was not found in the environment.")
@@ -63,12 +63,8 @@ class GLUETransformer(BaseTransformer):
                    examples,
                    self.tokenizer,
                    max_length=args.max_seq_length,
-                    task=args.task,
                    label_list=self.labels,
                    output_mode=args.glue_output_mode,
-                    pad_on_left=bool(args.model_type in ["xlnet"]),  # pad on the left for xlnet
-                    pad_token=self.tokenizer.convert_tokens_to_ids([self.tokenizer.pad_token])[0],
-                    pad_token_segment_id=self.tokenizer.pad_token_type_id,
                )
                logger.info("Saving features into cached file %s", cached_features_file)
                torch.save(features, cached_features_file)
@@ -5,4 +5,5 @@ seqeval
 psutil
 sacrebleu
 rouge-score
-tensorflow_datasets
+tensorflow_datasets
+pytorch-lightning==0.7.3  # April 10, 2020 release
@@ -22,6 +22,8 @@ import json
 import logging
 import os
 import random
+from dataclasses import dataclass, field
+from typing import Optional

 import numpy as np
 import torch
@@ -36,6 +38,8 @@ from transformers import (
    AutoConfig,
    AutoModelForSequenceClassification,
    AutoTokenizer,
+    HfArgumentParser,
+    TrainingArguments,
    get_linear_schedule_with_warmup,
 )
 from transformers import glue_compute_metrics as compute_metrics
@@ -158,7 +162,7 @@ def train(args, train_dataset, model, tokenizer):
    train_iterator = trange(
        epochs_trained, int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0],
    )
-    set_seed(args)  # Added here for reproductibility
+    set_seed(args)  # Added here for reproducibility
    for _ in train_iterator:
        epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
        for step, batch in enumerate(epoch_iterator):
@@ -354,14 +358,7 @@ def load_and_cache_examples(args, task, tokenizer, evaluate=False):
            processor.get_dev_examples(args.data_dir) if evaluate else processor.get_train_examples(args.data_dir)
        )
        features = convert_examples_to_features(
-            examples,
-            tokenizer,
-            label_list=label_list,
-            max_length=args.max_seq_length,
-            output_mode=output_mode,
-            pad_on_left=bool(args.model_type in ["xlnet"]),  # pad on the left for xlnet
-            pad_token=tokenizer.pad_token_id,
-            pad_token_segment_id=tokenizer.pad_token_type_id,
+            examples, tokenizer, max_length=args.max_seq_length, label_list=label_list, output_mode=output_mode,
        )
        if args.local_rank in [-1, 0]:
            logger.info("Saving features into cached file %s", cached_features_file)
@@ -383,137 +380,54 @@ def load_and_cache_examples(args, task, tokenizer, evaluate=False):
    return dataset


-def main():
-    parser = argparse.ArgumentParser()
+@dataclass
+class ModelArguments:
+    """
+    Arguments pertaining to which model/config/tokenizer we are going to fine-tune from.
+    """

-    # Required parameters
-    parser.add_argument(
-        "--data_dir",
-        default=None,
-        type=str,
-        required=True,
-        help="The input data dir. Should contain the .tsv files (or other data files) for the task.",
+    model_name_or_path: str = field(
+        metadata={"help": "Path to pre-trained model or shortcut name selected in the list: " + ", ".join(ALL_MODELS)}
    )
-    parser.add_argument(
-        "--model_type",
-        default=None,
-        type=str,
-        required=True,
-        help="Model type selected in the list: " + ", ".join(MODEL_TYPES),
+    model_type: str = field(metadata={"help": "Model type selected in the list: " + ", ".join(MODEL_TYPES)})
+    config_name: Optional[str] = field(
+        default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"}
    )
-    parser.add_argument(
-        "--model_name_or_path",
-        default=None,
-        type=str,
-        required=True,
-        help="Path to pre-trained model or shortcut name selected in the list: " + ", ".join(ALL_MODELS),
+    tokenizer_name: Optional[str] = field(
+        default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}
    )
-    parser.add_argument(
-        "--task_name",
-        default=None,
-        type=str,
-        required=True,
-        help="The name of the task to train selected in the list: " + ", ".join(processors.keys()),
-    )
-    parser.add_argument(
-        "--output_dir",
-        default=None,
-        type=str,
-        required=True,
-        help="The output directory where the model predictions and checkpoints will be written.",
+    cache_dir: Optional[str] = field(
+        default=None, metadata={"help": "Where do you want to store the pre-trained models downloaded from s3"}
    )

-    # Other parameters
-    parser.add_argument(
-        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name",
+
+@dataclass
+class DataProcessingArguments:
+    task_name: str = field(
+        metadata={"help": "The name of the task to train selected in the list: " + ", ".join(processors.keys())}
    )
-    parser.add_argument(
-        "--tokenizer_name",
-        default="",
-        type=str,
-        help="Pretrained tokenizer name or path if not the same as model_name",
+    data_dir: str = field(
+        metadata={"help": "The input data dir. Should contain the .tsv files (or other data files) for the task."}
    )
-    parser.add_argument(
-        "--cache_dir",
-        default="",
-        type=str,
-        help="Where do you want to store the pre-trained models downloaded from s3",
-    )
-    parser.add_argument(
-        "--max_seq_length",
+    max_seq_length: int = field(
        default=128,
-        type=int,
-        help="The maximum total input sequence length after tokenization. Sequences longer "
-        "than this will be truncated, sequences shorter will be padded.",
+        metadata={
+            "help": "The maximum total input sequence length after tokenization. Sequences longer "
+            "than this will be truncated, sequences shorter will be padded."
+        },
    )
-    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
-    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
-    parser.add_argument(
-        "--evaluate_during_training", action="store_true", help="Run evaluation during training at each logging step.",
-    )
-    parser.add_argument(
-        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model.",
+    overwrite_cache: bool = field(
+        default=False, metadata={"help": "Overwrite the cached training and evaluation sets"}
    )

-    parser.add_argument(
-        "--per_gpu_train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training.",
-    )
-    parser.add_argument(
-        "--per_gpu_eval_batch_size", default=8, type=int, help="Batch size per GPU/CPU for evaluation.",
-    )
-    parser.add_argument(
-        "--gradient_accumulation_steps",
-        type=int,
-        default=1,
-        help="Number of updates steps to accumulate before performing a backward/update pass.",
-    )
-    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
-    parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
-    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
-    parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.")
-    parser.add_argument(
-        "--num_train_epochs", default=3.0, type=float, help="Total number of training epochs to perform.",
-    )
-    parser.add_argument(
-        "--max_steps",
-        default=-1,
-        type=int,
-        help="If > 0: set total number of training steps to perform. Override num_train_epochs.",
-    )
-    parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")

-    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
-    parser.add_argument(
-        "--eval_all_checkpoints",
-        action="store_true",
-        help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number",
-    )
-    parser.add_argument("--no_cuda", action="store_true", help="Avoid using CUDA when available")
-    parser.add_argument(
-        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory",
-    )
-    parser.add_argument(
-        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets",
-    )
-    parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")
+def main():
+    parser = HfArgumentParser((ModelArguments, DataProcessingArguments, TrainingArguments))
+    model_args, dataprocessing_args, training_args = parser.parse_args_into_dataclasses()

-    parser.add_argument(
-        "--fp16",
-        action="store_true",
-        help="Whether to use 16-bit (mixed) precision (through NVIDIA apex) instead of 32-bit",
-    )
-    parser.add_argument(
-        "--fp16_opt_level",
-        type=str,
-        default="O1",
-        help="For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']."
-        "See details at https://nvidia.github.io/apex/amp.html",
-    )
-    parser.add_argument("--local_rank", type=int, default=-1, help="For distributed training: local_rank")
-    parser.add_argument("--server_ip", type=str, default="", help="For distant debugging.")
-    parser.add_argument("--server_port", type=str, default="", help="For distant debugging.")
-    args = parser.parse_args()
+    # For now, let's merge all the sets of args into one,
+    # but soon, we'll keep distinct sets of args, with a cleaner separation of concerns.
+    args = argparse.Namespace(**vars(model_args), **vars(dataprocessing_args), **vars(training_args))

    if (
        os.path.exists(args.output_dir)
@@ -522,20 +436,9 @@ def main():
        and not args.overwrite_output_dir
    ):
        raise ValueError(
-            "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
-                args.output_dir
-            )
+            f"Output directory ({args.output_dir}) already exists and is not empty. Use --overwrite_output_dir to overcome."
        )

-    # Setup distant debugging if needed
-    if args.server_ip and args.server_port:
-        # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
-        import ptvsd
-
-        print("Waiting for debugger attach")
-        ptvsd.enable_attach(address=(args.server_ip, args.server_port), redirect_output=True)
-        ptvsd.wait_for_attach()
-
    # Setup CUDA, GPU & distributed training
    if args.local_rank == -1 or args.no_cuda:
        device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
@@ -583,18 +486,16 @@ def main():
        args.config_name if args.config_name else args.model_name_or_path,
        num_labels=num_labels,
        finetuning_task=args.task_name,
-        cache_dir=args.cache_dir if args.cache_dir else None,
+        cache_dir=args.cache_dir,
    )
    tokenizer = AutoTokenizer.from_pretrained(
-        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
-        do_lower_case=args.do_lower_case,
-        cache_dir=args.cache_dir if args.cache_dir else None,
+        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path, cache_dir=args.cache_dir,
    )
    model = AutoModelForSequenceClassification.from_pretrained(
        args.model_name_or_path,
        from_tf=bool(".ckpt" in args.model_name_or_path),
        config=config,
-        cache_dir=args.cache_dir if args.cache_dir else None,
+        cache_dir=args.cache_dir,
    )

    if args.local_rank == 0:
@@ -636,7 +537,7 @@ def main():
    # Evaluation
    results = {}
    if args.do_eval and args.local_rank in [-1, 0]:
-        tokenizer = AutoTokenizer.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
+        tokenizer = AutoTokenizer.from_pretrained(args.output_dir)
        checkpoints = [args.output_dir]
        if args.eval_all_checkpoints:
            checkpoints = list(
@@ -67,7 +67,7 @@ class TextDataset(Dataset):
    def __init__(self, tokenizer: PreTrainedTokenizer, args, file_path: str, block_size=512):
        assert os.path.isfile(file_path)

-        block_size = block_size - (tokenizer.max_len - tokenizer.max_len_single_sentence)
+        block_size = block_size - tokenizer.num_special_tokens_to_add(pair=False)

        directory, filename = os.path.split(file_path)
        cached_features_file = os.path.join(
@@ -233,6 +233,9 @@ def train(args, train_dataset, model: PreTrainedModel, tokenizer: PreTrainedToke
    else:
        t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs

+    model = model.module if hasattr(model, "module") else model  # Take care of distributed/parallel training
+    model.resize_token_embeddings(len(tokenizer))
+
    # Prepare optimizer and schedule (linear warmup and decay)
    no_decay = ["bias", "LayerNorm.weight"]
    optimizer_grouped_parameters = [
@@ -309,16 +312,17 @@ def train(args, train_dataset, model: PreTrainedModel, tokenizer: PreTrainedToke

    tr_loss, logging_loss = 0.0, 0.0

-    model_to_resize = model.module if hasattr(model, "module") else model  # Take care of distributed/parallel training
-    model_to_resize.resize_token_embeddings(len(tokenizer))
-
    model.zero_grad()
    train_iterator = trange(
        epochs_trained, int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0]
    )
    set_seed(args)  # Added here for reproducibility
-    for _ in train_iterator:
+    for epoch in train_iterator:
        epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
+
+        if args.local_rank != -1:
+            train_sampler.set_epoch(epoch)
+
        for step, batch in enumerate(epoch_iterator):

            # Skip past any already trained steps if resuming training
@@ -624,6 +628,7 @@ def main():
        and os.listdir(args.output_dir)
        and args.do_train
        and not args.overwrite_output_dir
+        and not args.should_continue
    ):
        raise ValueError(
            "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
@@ -361,6 +361,7 @@ def load_and_cache_examples(args, task, tokenizer, evaluate=False, test=False):
            args.max_seq_length,
            tokenizer,
            pad_on_left=bool(args.model_type in ["xlnet"]),  # pad on the left for xlnet
+            pad_token=tokenizer.pad_token_id,
            pad_token_segment_id=tokenizer.pad_token_type_id,
        )
        if args.local_rank in [-1, 0]:
@@ -48,10 +48,10 @@ train_examples = info.splits["train"].num_examples
 valid_examples = info.splits["validation"].num_examples

 # Prepare dataset for GLUE as a tf.data.Dataset instance
-train_dataset = glue_convert_examples_to_features(data["train"], tokenizer, 128, TASK)
+train_dataset = glue_convert_examples_to_features(data["train"], tokenizer, max_length=128, task=TASK)

 # MNLI expects either validation_matched or validation_mismatched
-valid_dataset = glue_convert_examples_to_features(data["validation"], tokenizer, 128, TASK)
+valid_dataset = glue_convert_examples_to_features(data["validation"], tokenizer, max_length=128, task=TASK)
 train_dataset = train_dataset.shuffle(128).batch(BATCH_SIZE).repeat(-1)
 valid_dataset = valid_dataset.batch(EVAL_BATCH_SIZE)

@@ -0,0 +1,611 @@
+# coding=utf-8
+# Copyright 2019 The Google AI Language Team Authors and The HuggingFace Inc. team.
+# Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Finetuning the library models for sequence classification on GLUE (Bert, DistilBert, XLNet, RoBERTa)."""
+
+from __future__ import absolute_import, division, print_function
+
+import argparse
+import glob
+import logging
+import os
+import random
+
+import numpy as np
+import torch
+import torch_xla.core.xla_model as xm
+import torch_xla.debug.metrics as met
+import torch_xla.distributed.parallel_loader as pl
+import torch_xla.distributed.xla_multiprocessing as xmp
+from torch.utils.data import DataLoader, RandomSampler, TensorDataset
+from torch.utils.data.distributed import DistributedSampler
+from tqdm import tqdm, trange
+
+from transformers import (
+    WEIGHTS_NAME,
+    AdamW,
+    BertConfig,
+    BertForSequenceClassification,
+    BertTokenizer,
+    DistilBertConfig,
+    DistilBertForSequenceClassification,
+    DistilBertTokenizer,
+    RobertaConfig,
+    RobertaForSequenceClassification,
+    RobertaTokenizer,
+    XLMConfig,
+    XLMForSequenceClassification,
+    XLMTokenizer,
+    XLNetConfig,
+    XLNetForSequenceClassification,
+    XLNetTokenizer,
+    get_linear_schedule_with_warmup,
+)
+from transformers import glue_compute_metrics as compute_metrics
+from transformers import glue_convert_examples_to_features as convert_examples_to_features
+from transformers import glue_output_modes as output_modes
+from transformers import glue_processors as processors
+
+
+try:
+    # Only tensorboardX supports writing directly to gs://
+    from tensorboardX import SummaryWriter
+except ImportError:
+    from torch.utils.tensorboard import SummaryWriter
+
+
+logger = logging.getLogger(__name__)
+
+ALL_MODELS = sum(
+    (
+        tuple(conf.pretrained_config_archive_map.keys())
+        for conf in (BertConfig, XLNetConfig, XLMConfig, RobertaConfig, DistilBertConfig)
+    ),
+    (),
+)
+
+MODEL_CLASSES = {
+    "bert": (BertConfig, BertForSequenceClassification, BertTokenizer),
+    "xlnet": (XLNetConfig, XLNetForSequenceClassification, XLNetTokenizer),
+    "xlm": (XLMConfig, XLMForSequenceClassification, XLMTokenizer),
+    "roberta": (RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer),
+    "distilbert": (DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer),
+}
+
+
+def set_seed(seed):
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+
+
+def get_sampler(dataset):
+    if xm.xrt_world_size() <= 1:
+        return RandomSampler(dataset)
+    return DistributedSampler(dataset, num_replicas=xm.xrt_world_size(), rank=xm.get_ordinal())
+
+
+def train(args, train_dataset, model, tokenizer, disable_logging=False):
+    """ Train the model """
+    if xm.is_master_ordinal():
+        # Only master writes to Tensorboard
+        tb_writer = SummaryWriter(args.tensorboard_logdir)
+
+    train_sampler = get_sampler(train_dataset)
+    dataloader = DataLoader(train_dataset, sampler=train_sampler, batch_size=args.train_batch_size)
+
+    if args.max_steps > 0:
+        t_total = args.max_steps
+        args.num_train_epochs = args.max_steps // (len(dataloader) // args.gradient_accumulation_steps) + 1
+    else:
+        t_total = len(dataloader) // args.gradient_accumulation_steps * args.num_train_epochs
+
+    # Prepare optimizer and schedule (linear warmup and decay)
+    no_decay = ["bias", "LayerNorm.weight"]
+    optimizer_grouped_parameters = [
+        {
+            "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
+            "weight_decay": args.weight_decay,
+        },
+        {"params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], "weight_decay": 0.0},
+    ]
+    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)
+    scheduler = get_linear_schedule_with_warmup(
+        optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total,
+    )
+
+    # Train!
+    logger.info("***** Running training *****")
+    logger.info("  Num examples = %d", len(dataloader) * args.train_batch_size)
+    logger.info("  Num Epochs = %d", args.num_train_epochs)
+    logger.info("  Instantaneous batch size per TPU core = %d", args.train_batch_size)
+    logger.info(
+        "  Total train batch size (w. parallel, distributed & accumulation) = %d",
+        (args.train_batch_size * args.gradient_accumulation_steps * xm.xrt_world_size()),
+    )
+    logger.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
+    logger.info("  Total optimization steps = %d", t_total)
+
+    global_step = 0
+    loss = None
+    model.zero_grad()
+    train_iterator = trange(int(args.num_train_epochs), desc="Epoch", disable=disable_logging)
+    set_seed(args.seed)  # Added here for reproductibility (even between python 2 and 3)
+    for epoch in train_iterator:
+        # tpu-comment: Get TPU parallel loader which sends data to TPU in background.
+        train_dataloader = pl.ParallelLoader(dataloader, [args.device]).per_device_loader(args.device)
+        epoch_iterator = tqdm(train_dataloader, desc="Iteration", total=len(dataloader), disable=disable_logging)
+        for step, batch in enumerate(epoch_iterator):
+
+            # Save model checkpoint.
+            if args.save_steps > 0 and global_step % args.save_steps == 0:
+                output_dir = os.path.join(args.output_dir, "checkpoint-{}".format(global_step))
+                logger.info("Saving model checkpoint to %s", output_dir)
+
+                if xm.is_master_ordinal():
+                    if not os.path.exists(output_dir):
+                        os.makedirs(output_dir)
+                    torch.save(args, os.path.join(output_dir, "training_args.bin"))
+
+                # Barrier to wait for saving checkpoint.
+                xm.rendezvous("mid_training_checkpoint")
+                # model.save_pretrained needs to be called by all ordinals
+                model.save_pretrained(output_dir)
+
+            model.train()
+            inputs = {"input_ids": batch[0], "attention_mask": batch[1], "labels": batch[3]}
+            if args.model_type != "distilbert":
+                # XLM, DistilBERT and RoBERTa don't use segment_ids
+                inputs["token_type_ids"] = batch[2] if args.model_type in ["bert", "xlnet"] else None
+            outputs = model(**inputs)
+            loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
+
+            if args.gradient_accumulation_steps > 1:
+                loss = loss / args.gradient_accumulation_steps
+
+            loss.backward()
+
+            if (step + 1) % args.gradient_accumulation_steps == 0:
+                torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)
+
+                xm.optimizer_step(optimizer)
+                scheduler.step()  # Update learning rate schedule
+                model.zero_grad()
+                global_step += 1
+
+                if args.logging_steps > 0 and global_step % args.logging_steps == 0:
+                    # Log metrics.
+                    results = {}
+                    if args.evaluate_during_training:
+                        results = evaluate(args, model, tokenizer, disable_logging=disable_logging)
+                    loss_scalar = loss.item()
+                    logger.info(
+                        "global_step: {global_step}, lr: {lr:.6f}, loss: {loss:.3f}".format(
+                            global_step=global_step, lr=scheduler.get_lr()[0], loss=loss_scalar
+                        )
+                    )
+                    if xm.is_master_ordinal():
+                        # tpu-comment: All values must be in CPU and not on TPU device
+                        for key, value in results.items():
+                            tb_writer.add_scalar("eval_{}".format(key), value, global_step)
+                        tb_writer.add_scalar("lr", scheduler.get_lr()[0], global_step)
+                        tb_writer.add_scalar("loss", loss_scalar, global_step)
+
+            if args.max_steps > 0 and global_step > args.max_steps:
+                epoch_iterator.close()
+                break
+        if args.metrics_debug:
+            # tpu-comment: Logging debug metrics for PyTorch/XLA (compile, execute times, ops, etc.)
+            xm.master_print(met.metrics_report())
+        if args.max_steps > 0 and global_step > args.max_steps:
+            train_iterator.close()
+            break
+
+    if xm.is_master_ordinal():
+        tb_writer.close()
+    return global_step, loss.item()
+
+
+def evaluate(args, model, tokenizer, prefix="", disable_logging=False):
+    """Evaluate the model"""
+    if xm.is_master_ordinal():
+        # Only master writes to Tensorboard
+        tb_writer = SummaryWriter(args.tensorboard_logdir)
+
+    # Loop to handle MNLI double evaluation (matched, mis-matched)
+    eval_task_names = ("mnli", "mnli-mm") if args.task_name == "mnli" else (args.task_name,)
+    eval_outputs_dirs = (args.output_dir, args.output_dir + "-MM") if args.task_name == "mnli" else (args.output_dir,)
+
+    results = {}
+    for eval_task, eval_output_dir in zip(eval_task_names, eval_outputs_dirs):
+        eval_dataset = load_and_cache_examples(args, eval_task, tokenizer, evaluate=True)
+        eval_sampler = get_sampler(eval_dataset)
+
+        if not os.path.exists(eval_output_dir):
+            os.makedirs(eval_output_dir)
+
+        dataloader = DataLoader(eval_dataset, sampler=eval_sampler, batch_size=args.eval_batch_size, shuffle=False)
+        eval_dataloader = pl.ParallelLoader(dataloader, [args.device]).per_device_loader(args.device)
+
+        # Eval!
+        logger.info("***** Running evaluation {} *****".format(prefix))
+        logger.info("  Num examples = %d", len(dataloader) * args.eval_batch_size)
+        logger.info("  Batch size = %d", args.eval_batch_size)
+        eval_loss = 0.0
+        nb_eval_steps = 0
+        preds = None
+        out_label_ids = None
+        for batch in tqdm(eval_dataloader, desc="Evaluating", disable=disable_logging):
+            model.eval()
+
+            with torch.no_grad():
+                inputs = {"input_ids": batch[0], "attention_mask": batch[1], "labels": batch[3]}
+                if args.model_type != "distilbert":
+                    # XLM, DistilBERT and RoBERTa don't use segment_ids
+                    inputs["token_type_ids"] = batch[2] if args.model_type in ["bert", "xlnet"] else None
+                outputs = model(**inputs)
+                batch_eval_loss, logits = outputs[:2]
+
+                eval_loss += batch_eval_loss
+            nb_eval_steps += 1
+            if preds is None:
+                preds = logits.detach().cpu().numpy()
+                out_label_ids = inputs["labels"].detach().cpu().numpy()
+            else:
+                preds = np.append(preds, logits.detach().cpu().numpy(), axis=0)
+                out_label_ids = np.append(out_label_ids, inputs["labels"].detach().cpu().numpy(), axis=0)
+
+        # tpu-comment: Get all predictions and labels from all worker shards of eval dataset
+        preds = xm.mesh_reduce("eval_preds", preds, np.concatenate)
+        out_label_ids = xm.mesh_reduce("eval_out_label_ids", out_label_ids, np.concatenate)
+
+        eval_loss = eval_loss / nb_eval_steps
+        if args.output_mode == "classification":
+            preds = np.argmax(preds, axis=1)
+        elif args.output_mode == "regression":
+            preds = np.squeeze(preds)
+        result = compute_metrics(eval_task, preds, out_label_ids)
+        results.update(result)
+        results["eval_loss"] = eval_loss.item()
+
+        output_eval_file = os.path.join(eval_output_dir, prefix, "eval_results.txt")
+        if xm.is_master_ordinal():
+            with open(output_eval_file, "w") as writer:
+                logger.info("***** Eval results {} *****".format(prefix))
+                for key in sorted(results.keys()):
+                    logger.info("  %s = %s", key, str(results[key]))
+                    writer.write("%s = %s\n" % (key, str(results[key])))
+                    tb_writer.add_scalar(f"{eval_task}/{key}", results[key])
+
+    if args.metrics_debug:
+        # tpu-comment: Logging debug metrics for PyTorch/XLA (compile, execute times, ops, etc.)
+        xm.master_print(met.metrics_report())
+
+    if xm.is_master_ordinal():
+        tb_writer.close()
+
+    return results
+
+
+def load_and_cache_examples(args, task, tokenizer, evaluate=False):
+    if not xm.is_master_ordinal():
+        xm.rendezvous("load_and_cache_examples")
+
+    processor = processors[task]()
+    output_mode = output_modes[task]
+    cached_features_file = os.path.join(
+        args.cache_dir,
+        "cached_{}_{}_{}_{}".format(
+            "dev" if evaluate else "train",
+            list(filter(None, args.model_name_or_path.split("/"))).pop(),
+            str(args.max_seq_length),
+            str(task),
+        ),
+    )
+
+    # Load data features from cache or dataset file
+    if os.path.exists(cached_features_file) and not args.overwrite_cache:
+        logger.info("Loading features from cached file %s", cached_features_file)
+        features = torch.load(cached_features_file)
+    else:
+        logger.info("Creating features from dataset file at %s", args.data_dir)
+        label_list = processor.get_labels()
+        if task in ["mnli", "mnli-mm"] and args.model_type in ["roberta"]:
+            # HACK(label indices are swapped in RoBERTa pretrained model)
+            label_list[1], label_list[2] = label_list[2], label_list[1]
+        examples = (
+            processor.get_dev_examples(args.data_dir) if evaluate else processor.get_train_examples(args.data_dir)
+        )
+        features = convert_examples_to_features(
+            examples, tokenizer, max_length=args.max_seq_length, label_list=label_list, output_mode=output_mode,
+        )
+        logger.info("Saving features into cached file %s", cached_features_file)
+        torch.save(features, cached_features_file)
+
+    if xm.is_master_ordinal():
+        xm.rendezvous("load_and_cache_examples")
+
+    # Convert to Tensors and build dataset
+    all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
+    all_attention_mask = torch.tensor([f.attention_mask for f in features], dtype=torch.long)
+    all_token_type_ids = torch.tensor([f.token_type_ids for f in features], dtype=torch.long)
+    if output_mode == "classification":
+        all_labels = torch.tensor([f.label for f in features], dtype=torch.long)
+    elif output_mode == "regression":
+        all_labels = torch.tensor([f.label for f in features], dtype=torch.float)
+
+    dataset = TensorDataset(all_input_ids, all_attention_mask, all_token_type_ids, all_labels)
+    return dataset
+
+
+def main(args):
+    if (
+        os.path.exists(args.output_dir)
+        and os.listdir(args.output_dir)
+        and args.do_train
+        and not args.overwrite_output_dir
+    ):
+        raise ValueError(
+            (
+                "Output directory ({}) already exists and is not empty." " Use --overwrite_output_dir to overcome."
+            ).format(args.output_dir)
+        )
+
+    # tpu-comment: Get TPU/XLA Device
+    args.device = xm.xla_device()
+
+    # Setup logging
+    logging.basicConfig(
+        format="[xla:{}] %(asctime)s - %(levelname)s - %(name)s -   %(message)s".format(xm.get_ordinal()),
+        datefmt="%m/%d/%Y %H:%M:%S",
+        level=logging.INFO,
+    )
+    disable_logging = False
+    if not xm.is_master_ordinal() and args.only_log_master:
+        # Disable all non-master loggers below CRITICAL.
+        logging.disable(logging.CRITICAL)
+        disable_logging = True
+    logger.warning("Process rank: %s, device: %s, num_cores: %s", xm.get_ordinal(), args.device, args.num_cores)
+
+    # Set seed to have same initialization
+    set_seed(args.seed)
+
+    # Prepare GLUE task
+    args.task_name = args.task_name.lower()
+    if args.task_name not in processors:
+        raise ValueError("Task not found: %s" % (args.task_name))
+    processor = processors[args.task_name]()
+    args.output_mode = output_modes[args.task_name]
+    label_list = processor.get_labels()
+    num_labels = len(label_list)
+
+    if not xm.is_master_ordinal():
+        xm.rendezvous(
+            "download_only_once"
+        )  # Make sure only the first process in distributed training will download model & vocab
+
+    # Load pretrained model and tokenizer
+    args.model_type = args.model_type.lower()
+    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
+    config = config_class.from_pretrained(
+        args.config_name if args.config_name else args.model_name_or_path,
+        num_labels=num_labels,
+        finetuning_task=args.task_name,
+        cache_dir=args.cache_dir if args.cache_dir else None,
+        xla_device=True,
+    )
+    tokenizer = tokenizer_class.from_pretrained(
+        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
+        do_lower_case=args.do_lower_case,
+        cache_dir=args.cache_dir if args.cache_dir else None,
+    )
+    model = model_class.from_pretrained(
+        args.model_name_or_path,
+        from_tf=bool(".ckpt" in args.model_name_or_path),
+        config=config,
+        cache_dir=args.cache_dir if args.cache_dir else None,
+    )
+
+    if xm.is_master_ordinal():
+        xm.rendezvous("download_only_once")
+
+    # Send model to TPU/XLA device.
+    model.to(args.device)
+
+    logger.info("Training/evaluation parameters %s", args)
+
+    if args.do_train:
+        # Train the model.
+        train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False)
+        global_step, tr_loss = train(args, train_dataset, model, tokenizer, disable_logging=disable_logging)
+        logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)
+
+        if xm.is_master_ordinal():
+            # Save trained model.
+            # Saving best-practices: if you use defaults names for the model, you can reload it using from_pretrained()
+
+            # Create output directory if needed
+            if not os.path.exists(args.output_dir):
+                os.makedirs(args.output_dir)
+
+            logger.info("Saving model checkpoint to %s", args.output_dir)
+            # Save a trained model, configuration and tokenizer using `save_pretrained()`.
+            # They can then be reloaded using `from_pretrained()`
+            tokenizer.save_pretrained(args.output_dir)
+            # Good practice: save your training arguments together with the trained.
+            torch.save(args, os.path.join(args.output_dir, "training_args.bin"))
+
+        xm.rendezvous("post_training_checkpoint")
+        # model.save_pretrained needs to be called by all ordinals
+        model.save_pretrained(args.output_dir)
+
+        # Load a trained model and vocabulary that you have fine-tuned
+        model = model_class.from_pretrained(args.output_dir)
+        tokenizer = tokenizer_class.from_pretrained(args.output_dir)
+        model.to(args.device)
+
+    # Evaluation
+    results = {}
+    if args.do_eval:
+        tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
+        checkpoints = [args.output_dir]
+        if args.eval_all_checkpoints:
+            checkpoints = list(
+                os.path.dirname(c) for c in sorted(glob.glob(args.output_dir + "/**/" + WEIGHTS_NAME, recursive=True))
+            )
+            logging.getLogger("transformers.modeling_utils").setLevel(logging.WARN)  # Reduce logging
+        logger.info("Evaluate the following checkpoints: %s", checkpoints)
+        for checkpoint in checkpoints:
+            global_step = checkpoint.split("-")[-1] if len(checkpoints) > 1 else ""
+            prefix = checkpoint.split("/")[-1] if checkpoint.find("checkpoint") != -1 else ""
+
+            model = model_class.from_pretrained(checkpoint)
+            model.to(args.device)
+            result = evaluate(args, model, tokenizer, prefix=prefix, disable_logging=disable_logging)
+            result = dict((k + "_{}".format(global_step), v) for k, v in result.items())
+            results.update(result)
+
+    return results
+
+
+def get_args():
+    parser = argparse.ArgumentParser()
+
+    # Required parameters
+    parser.add_argument(
+        "--data_dir",
+        default=None,
+        type=str,
+        required=True,
+        help="The input data dir. Should contain the .tsv files (or other data files) for the task.",
+    )
+    parser.add_argument(
+        "--model_type",
+        default=None,
+        type=str,
+        required=True,
+        help="Model type selected in the list: " + ", ".join(MODEL_CLASSES.keys()),
+    )
+    parser.add_argument(
+        "--model_name_or_path",
+        default=None,
+        type=str,
+        required=True,
+        help="Path to pre-trained model or shortcut name selected in the list: " + ", ".join(ALL_MODELS),
+    )
+    parser.add_argument(
+        "--task_name",
+        default=None,
+        type=str,
+        required=True,
+        help="The name of the task to train selected in the list: " + ", ".join(processors.keys()),
+    )
+    parser.add_argument(
+        "--output_dir",
+        default=None,
+        type=str,
+        required=True,
+        help="The output directory where the model predictions and checkpoints will be written.",
+    )
+
+    # TPU Parameters
+    parser.add_argument("--num_cores", default=8, type=int, help="Number of TPU cores to use (1 or 8).")
+    parser.add_argument("--metrics_debug", action="store_true", help="Whether to print debug metrics.")
+
+    # Other parameters
+    parser.add_argument(
+        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
+    )
+    parser.add_argument(
+        "--tokenizer_name",
+        default="",
+        type=str,
+        help="Pretrained tokenizer name or path if not the same as model_name",
+    )
+    parser.add_argument(
+        "--cache_dir",
+        default="",
+        type=str,
+        help="Where do you want to store the pre-trained models downloaded and features file generated",
+    )
+    parser.add_argument(
+        "--max_seq_length",
+        default=128,
+        type=int,
+        help="The maximum total input sequence length after tokenization. Sequences longer "
+        "than this will be truncated, sequences shorter will be padded.",
+    )
+    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
+    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
+    parser.add_argument(
+        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step."
+    )
+    parser.add_argument(
+        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model."
+    )
+
+    parser.add_argument("--train_batch_size", default=8, type=int, help="Per core batch size for training.")
+    parser.add_argument("--eval_batch_size", default=8, type=int, help="Per core batch size for evaluation.")
+    parser.add_argument(
+        "--gradient_accumulation_steps",
+        type=int,
+        default=1,
+        help="Number of updates steps to accumulate before performing a backward/update pass.",
+    )
+    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
+    parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight deay if we apply some.")
+    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
+    parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.")
+    parser.add_argument(
+        "--num_train_epochs", default=3.0, type=float, help="Total number of training epochs to perform."
+    )
+    parser.add_argument(
+        "--max_steps",
+        default=-1,
+        type=int,
+        help="If > 0: set total number of training steps to perform. Override num_train_epochs.",
+    )
+    parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
+    parser.add_argument("--tensorboard_logdir", default="./runs", type=str, help="Where to write tensorboard metrics.")
+    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X update steps.")
+    parser.add_argument("--only_log_master", action="store_true", help="Whether to log only from each hosts master.")
+    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X update steps.")
+    parser.add_argument(
+        "--eval_all_checkpoints",
+        action="store_true",
+        help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number",
+    )
+    parser.add_argument(
+        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory"
+    )
+    parser.add_argument(
+        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets"
+    )
+    parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")
+
+    return parser.parse_args()
+
+
+def _mp_fn(rank, args):
+    main(args)
+
+
+def main_cli():
+    args = get_args()
+    xmp.spawn(_mp_fn, args=(args,), nprocs=args.num_cores)
+
+
+if __name__ == "__main__":
+    main_cli()
@@ -344,14 +344,7 @@ def load_and_cache_examples(args, task, tokenizer, evaluate=False):
            processor.get_test_examples(args.data_dir) if evaluate else processor.get_train_examples(args.data_dir)
        )
        features = convert_examples_to_features(
-            examples,
-            tokenizer,
-            label_list=label_list,
-            max_length=args.max_seq_length,
-            output_mode=output_mode,
-            pad_on_left=False,
-            pad_token=tokenizer.pad_token_id,
-            pad_token_segment_id=tokenizer.pad_token_type_id,
+            examples, tokenizer, max_length=args.max_seq_length, label_list=label_list, output_mode=output_mode,
        )
        if args.local_rank in [-1, 0]:
            logger.info("Saving features into cached file %s", cached_features_file)
@@ -20,7 +20,7 @@ def generate_summaries(
    examples: list, out_file: str, model_name: str, batch_size: int = 8, device: str = DEFAULT_DEVICE
 ):
    fout = Path(out_file).open("w")
-    model = BartForConditionalGeneration.from_pretrained(model_name, output_past=True,).to(device)
+    model = BartForConditionalGeneration.from_pretrained(model_name).to(device)
    tokenizer = BartTokenizer.from_pretrained("bart-large")

    max_length = 140
@@ -8,41 +8,41 @@ import torch
 from torch.utils.data import DataLoader

 from transformer_base import BaseTransformer, add_generic_args, generic_train, get_linear_schedule_with_warmup
-from utils import SummarizationDataset
+
+
+try:
+    from .utils import SummarizationDataset
+except ImportError:
+    from utils import SummarizationDataset


 logger = logging.getLogger(__name__)


-class BartSystem(BaseTransformer):
+class SummarizationTrainer(BaseTransformer):

    mode = "language-modeling"

    def __init__(self, hparams):
-        super(BartSystem, self).__init__(hparams, num_labels=None, mode=self.mode)
+        super().__init__(hparams, num_labels=None, mode=self.mode)
+        self.dataset_kwargs: dict = dict(
+            data_dir=self.hparams.data_dir,
+            max_source_length=self.hparams.max_source_length,
+            max_target_length=self.hparams.max_target_length,
+        )

-    def forward(
-        self, input_ids, attention_mask=None, decoder_input_ids=None, decoder_attention_mask=None, lm_labels=None
-    ):
+    def forward(self, input_ids, attention_mask=None, decoder_input_ids=None, lm_labels=None):
        return self.model(
-            input_ids,
-            attention_mask=attention_mask,
-            decoder_input_ids=decoder_input_ids,
-            decoder_attention_mask=decoder_attention_mask,
-            lm_labels=lm_labels,
+            input_ids, attention_mask=attention_mask, decoder_input_ids=decoder_input_ids, lm_labels=lm_labels,
        )

    def _step(self, batch):
-        y = batch["target_ids"]
+        pad_token_id = self.tokenizer.pad_token_id
+        source_ids, source_mask, y = batch["source_ids"], batch["source_mask"], batch["target_ids"]
        y_ids = y[:, :-1].contiguous()
        lm_labels = y[:, 1:].clone()
-        lm_labels[y[:, 1:] == self.tokenizer.pad_token_id] = -100
-        outputs = self(
-            input_ids=batch["source_ids"],
-            attention_mask=batch["source_mask"],
-            decoder_input_ids=y_ids,
-            lm_labels=lm_labels,
-        )
+        lm_labels[y[:, 1:] == pad_token_id] = -100
+        outputs = self(source_ids, attention_mask=source_mask, decoder_input_ids=y_ids, lm_labels=lm_labels,)

        loss = outputs[0]

@@ -64,23 +64,24 @@ class BartSystem(BaseTransformer):
        return {"avg_val_loss": avg_loss, "log": tensorboard_logs}

    def test_step(self, batch, batch_idx):
+        pad_token_id = self.tokenizer.pad_token_id
+        source_ids, source_mask, y = SummarizationDataset.trim_seq2seq_batch(batch, pad_token_id)
+        # NOTE: the following kwargs get more speed and lower quality summaries than those in evaluate_cnn.py
        generated_ids = self.model.generate(
-            batch["source_ids"],
-            attention_mask=batch["source_mask"],
+            input_ids=source_ids,
+            attention_mask=source_mask,
            num_beams=1,
            max_length=80,
            repetition_penalty=2.5,
            length_penalty=1.0,
            early_stopping=True,
+            use_cache=True,
        )
        preds = [
            self.tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True)
            for g in generated_ids
        ]
-        target = [
-            self.tokenizer.decode(t, skip_special_tokens=True, clean_up_tokenization_spaces=True)
-            for t in batch["target_ids"]
-        ]
+        target = [self.tokenizer.decode(t, skip_special_tokens=True, clean_up_tokenization_spaces=True) for t in y]
        loss = self._step(batch)

        return {"val_loss": loss, "preds": preds, "target": target}
@@ -101,11 +102,13 @@ class BartSystem(BaseTransformer):

        return self.test_end(outputs)

-    def train_dataloader(self):
-        train_dataset = SummarizationDataset(
-            self.tokenizer, data_dir=self.hparams.data_dir, type_path="train", block_size=self.hparams.max_seq_length
-        )
-        dataloader = DataLoader(train_dataset, batch_size=self.hparams.train_batch_size)
+    def get_dataloader(self, type_path: str, batch_size: int) -> DataLoader:
+        dataset = SummarizationDataset(self.tokenizer, type_path=type_path, **self.dataset_kwargs)
+        dataloader = DataLoader(dataset, batch_size=batch_size, collate_fn=dataset.collate_fn)
+        return dataloader
+
+    def train_dataloader(self) -> DataLoader:
+        dataloader = self.get_dataloader("train", batch_size=self.hparams.train_batch_size)
        t_total = (
            (len(dataloader.dataset) // (self.hparams.train_batch_size * max(1, self.hparams.n_gpu)))
            // self.hparams.gradient_accumulation_steps
@@ -117,29 +120,30 @@ class BartSystem(BaseTransformer):
        self.lr_scheduler = scheduler
        return dataloader

-    def val_dataloader(self):
-        val_dataset = SummarizationDataset(
-            self.tokenizer, data_dir=self.hparams.data_dir, type_path="val", block_size=self.hparams.max_seq_length
-        )
-        return DataLoader(val_dataset, batch_size=self.hparams.eval_batch_size)
+    def val_dataloader(self) -> DataLoader:
+        return self.get_dataloader("val", batch_size=self.hparams.eval_batch_size)

-    def test_dataloader(self):
-        test_dataset = SummarizationDataset(
-            self.tokenizer, data_dir=self.hparams.data_dir, type_path="test", block_size=self.hparams.max_seq_length
-        )
-        return DataLoader(test_dataset, batch_size=self.hparams.eval_batch_size)
+    def test_dataloader(self) -> DataLoader:
+        return self.get_dataloader("test", batch_size=self.hparams.eval_batch_size)

    @staticmethod
    def add_model_specific_args(parser, root_dir):
        BaseTransformer.add_model_specific_args(parser, root_dir)
        # Add BART specific options
        parser.add_argument(
-            "--max_seq_length",
+            "--max_source_length",
            default=1024,
            type=int,
            help="The maximum total input sequence length after tokenization. Sequences longer "
            "than this will be truncated, sequences shorter will be padded.",
        )
+        parser.add_argument(
+            "--max_target_length",
+            default=56,
+            type=int,
+            help="The maximum total input sequence length after tokenization. Sequences longer "
+            "than this will be truncated, sequences shorter will be padded.",
+        )

        parser.add_argument(
            "--data_dir",
@@ -151,22 +155,30 @@ class BartSystem(BaseTransformer):
        return parser


-if __name__ == "__main__":
-    parser = argparse.ArgumentParser()
-    add_generic_args(parser, os.getcwd())
-    parser = BartSystem.add_model_specific_args(parser, os.getcwd())
-    args = parser.parse_args()
+def main(args):

    # If output_dir not provided, a folder will be generated in pwd
-    if args.output_dir is None:
+    if not args.output_dir:
        args.output_dir = os.path.join("./results", f"{args.task}_{args.model_type}_{time.strftime('%Y%m%d_%H%M%S')}",)
        os.makedirs(args.output_dir)
-
-    model = BartSystem(args)
+    model = SummarizationTrainer(args)
    trainer = generic_train(model, args)

    # Optionally, predict on dev set and write to output_dir
    if args.do_predict:
+        # See https://github.com/huggingface/transformers/issues/3159
+        # pl use this format to create a checkpoint:
+        # https://github.com/PyTorchLightning/pytorch-lightning/blob/master\
+        # /pytorch_lightning/callbacks/model_checkpoint.py#L169
        checkpoints = list(sorted(glob.glob(os.path.join(args.output_dir, "checkpointepoch=*.ckpt"), recursive=True)))
-        BartSystem.load_from_checkpoint(checkpoints[-1])
+        model = model.load_from_checkpoint(checkpoints[-1])
        trainer.test(model)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    add_generic_args(parser, os.getcwd())
+    parser = SummarizationTrainer.add_model_specific_args(parser, os.getcwd())
+    args = parser.parse_args()
+
+    main(args)
@@ -1,7 +1,3 @@
-# Install newest ptl.
-pip install -U git+http://github.com/PyTorchLightning/pytorch-lightning/
-
-
 export OUTPUT_DIR_NAME=bart_sum
 export CURRENT_DIR=${PWD}
 export OUTPUT_DIR=${CURRENT_DIR}/${OUTPUT_DIR_NAME}
@@ -12,7 +8,7 @@ mkdir -p $OUTPUT_DIR
 # Add parent directory to python path to access transformer_base.py
 export PYTHONPATH="../../":"${PYTHONPATH}"

-python run_bart_sum.py \
+python finetune.py \
 --data_dir=./cnn-dailymail/cnn_dm \
 --model_type=bart \
 --model_name_or_path=bart-large \
@@ -20,4 +16,4 @@ python run_bart_sum.py \
 --train_batch_size=4 \
 --eval_batch_size=4 \
 --output_dir=$OUTPUT_DIR \
--do_train
+--do_train  $@
@@ -0,0 +1,33 @@
+# Script for verifying that run_bart_sum can be invoked from its directory
+
+# Get tiny dataset with cnn_dm format (4 examples for train, val, test)
+wget https://s3.amazonaws.com/datasets.huggingface.co/summarization/cnn_tiny.tgz
+tar -xzvf cnn_tiny.tgz
+rm cnn_tiny.tgz
+
+export OUTPUT_DIR_NAME=bart_utest_output
+export CURRENT_DIR=${PWD}
+export OUTPUT_DIR=${CURRENT_DIR}/${OUTPUT_DIR_NAME}
+
+# Make output directory if it doesn't exist
+mkdir -p $OUTPUT_DIR
+
+# Add parent directory to python path to access transformer_base.py and utils.py
+export PYTHONPATH="../../":"${PYTHONPATH}"
+python finetune.py \
+--data_dir=cnn_tiny/ \
+--model_type=bart \
+--model_name_or_path=sshleifer/bart-tiny-random \
+--learning_rate=3e-5 \
+--train_batch_size=2 \
+--eval_batch_size=2 \
+--output_dir=$OUTPUT_DIR \
+--num_train_epochs=1  \
+--n_gpu=0 \
+--do_train $@
+
+rm -rf cnn_tiny
+rm -rf $OUTPUT_DIR
+
+
+
@@ -1,32 +1,148 @@
+import argparse
 import logging
+import os
 import sys
 import tempfile
 import unittest
 from pathlib import Path
 from unittest.mock import patch

+from torch.utils.data import DataLoader
+
+from transformers import BartTokenizer
+
 from .evaluate_cnn import run_generate
+from .finetune import main
+from .utils import SummarizationDataset


-articles = [" New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County."]
-
 logging.basicConfig(level=logging.DEBUG)

 logger = logging.getLogger()

+DEFAULT_ARGS = {
+    "output_dir": "",
+    "fp16": False,
+    "fp16_opt_level": "O1",
+    "n_gpu": 1,
+    "n_tpu_cores": 0,
+    "max_grad_norm": 1.0,
+    "do_train": True,
+    "do_predict": False,
+    "gradient_accumulation_steps": 1,
+    "server_ip": "",
+    "server_port": "",
+    "seed": 42,
+    "model_type": "bart",
+    "model_name_or_path": "sshleifer/bart-tiny-random",
+    "config_name": "",
+    "tokenizer_name": "",
+    "cache_dir": "",
+    "do_lower_case": False,
+    "learning_rate": 3e-05,
+    "weight_decay": 0.0,
+    "adam_epsilon": 1e-08,
+    "warmup_steps": 0,
+    "num_train_epochs": 1,
+    "train_batch_size": 2,
+    "eval_batch_size": 2,
+    "max_source_length": 12,
+    "max_target_length": 12,
+}
+
+
+def _dump_articles(path: Path, articles: list):
+    with path.open("w") as f:
+        f.write("\n".join(articles))
+
+
+def make_test_data_dir():
+    tmp_dir = Path(tempfile.gettempdir())
+    articles = [" Sam ate lunch today", "Sams lunch ingredients"]
+    summaries = ["A very interesting story about what I ate for lunch.", "Avocado, celery, turkey, coffee"]
+    for split in ["train", "val", "test"]:
+        _dump_articles((tmp_dir / f"{split}.source"), articles)
+        _dump_articles((tmp_dir / f"{split}.target"), summaries)
+    return tmp_dir
+

 class TestBartExamples(unittest.TestCase):
-    def test_bart_cnn_cli(self):
+    @classmethod
+    def setUpClass(cls):
        stream_handler = logging.StreamHandler(sys.stdout)
        logger.addHandler(stream_handler)
+        logging.disable(logging.CRITICAL)  # remove noisy download output from tracebacks
+        return cls
+
+    def test_bart_cnn_cli(self):
        tmp = Path(tempfile.gettempdir()) / "utest_generations_bart_sum.hypo"
-        with tmp.open("w") as f:
-            f.write("\n".join(articles))
-
        output_file_name = Path(tempfile.gettempdir()) / "utest_output_bart_sum.hypo"
-
+        articles = [" New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County."]
+        _dump_articles(tmp, articles)
        testargs = ["evaluate_cnn.py", str(tmp), str(output_file_name), "sshleifer/bart-tiny-random"]
-
        with patch.object(sys, "argv", testargs):
            run_generate()
            self.assertTrue(Path(output_file_name).exists())
+            os.remove(Path(output_file_name))
+
+    def test_bart_run_sum_cli(self):
+        args_d: dict = DEFAULT_ARGS.copy()
+        tmp_dir = make_test_data_dir()
+        output_dir = tempfile.mkdtemp(prefix="output_")
+        args_d.update(
+            data_dir=tmp_dir, model_type="bart", train_batch_size=2, eval_batch_size=2, n_gpu=0, output_dir=output_dir,
+        )
+        main(argparse.Namespace(**args_d))
+        args_d.update({"do_train": False, "do_predict": True})
+
+        main(argparse.Namespace(**args_d))
+        contents = os.listdir(output_dir)
+        expected_contents = {
+            "checkpointepoch=0.ckpt",
+            "test_results.txt",
+        }
+        created_files = {os.path.basename(p) for p in contents}
+        self.assertSetEqual(expected_contents, created_files)
+
+    def test_t5_run_sum_cli(self):
+        args_d: dict = DEFAULT_ARGS.copy()
+        tmp_dir = make_test_data_dir()
+        output_dir = tempfile.mkdtemp(prefix="output_")
+        args_d.update(
+            data_dir=tmp_dir,
+            model_type="t5",
+            model_name_or_path="patrickvonplaten/t5-tiny-random",
+            train_batch_size=2,
+            eval_batch_size=2,
+            n_gpu=0,
+            output_dir=output_dir,
+            do_predict=True,
+        )
+        main(argparse.Namespace(**args_d))
+
+        # args_d.update({"do_train": False, "do_predict": True})
+        # main(argparse.Namespace(**args_d))
+
+    def test_bart_summarization_dataset(self):
+        tmp_dir = Path(tempfile.gettempdir())
+        articles = [" Sam ate lunch today", "Sams lunch ingredients"]
+        summaries = ["A very interesting story about what I ate for lunch.", "Avocado, celery, turkey, coffee"]
+        _dump_articles((tmp_dir / "train.source"), articles)
+        _dump_articles((tmp_dir / "train.target"), summaries)
+        tokenizer = BartTokenizer.from_pretrained("bart-large")
+        max_len_source = max(len(tokenizer.encode(a)) for a in articles)
+        max_len_target = max(len(tokenizer.encode(a)) for a in summaries)
+        trunc_target = 4
+        train_dataset = SummarizationDataset(
+            tokenizer, data_dir=tmp_dir, type_path="train", max_source_length=20, max_target_length=trunc_target,
+        )
+        dataloader = DataLoader(train_dataset, batch_size=2, collate_fn=train_dataset.collate_fn)
+        for batch in dataloader:
+            self.assertEqual(batch["source_mask"].shape, batch["source_ids"].shape)
+            # show that articles were trimmed.
+            self.assertEqual(batch["source_ids"].shape[1], max_len_source)
+            self.assertGreater(20, batch["source_ids"].shape[1])  # trimmed significantly
+
+            # show that targets were truncated
+            self.assertEqual(batch["target_ids"].shape[1], trunc_target)  # Truncated
+            self.assertGreater(max_len_target, trunc_target)  # Truncated
@@ -1,35 +1,35 @@
 import os

+import torch
 from torch.utils.data import Dataset

+from transformers.tokenization_utils import trim_batch
+
+
+def encode_file(tokenizer, data_path, max_length, pad_to_max_length=True, return_tensors="pt"):
+    examples = []
+    with open(data_path, "r") as f:
+        for text in f.readlines():
+            tokenized = tokenizer.batch_encode_plus(
+                [text], max_length=max_length, pad_to_max_length=pad_to_max_length, return_tensors=return_tensors,
+            )
+            examples.append(tokenized)
+    return examples
+

 class SummarizationDataset(Dataset):
-    def __init__(self, tokenizer, data_dir="./cnn-dailymail/cnn_dm/", type_path="train", block_size=1024):
-        super(SummarizationDataset,).__init__()
+    def __init__(
+        self,
+        tokenizer,
+        data_dir="./cnn-dailymail/cnn_dm/",
+        type_path="train",
+        max_source_length=1024,
+        max_target_length=56,
+    ):
+        super().__init__()
        self.tokenizer = tokenizer
-
-        self.source = []
-        self.target = []
-
-        print("loading " + type_path + " source.")
-
-        with open(os.path.join(data_dir, type_path + ".source"), "r") as f:
-            for text in f.readlines():  # each text is a line and a full story
-                tokenized = tokenizer.batch_encode_plus(
-                    [text], max_length=block_size, pad_to_max_length=True, return_tensors="pt"
-                )
-                self.source.append(tokenized)
-            f.close()
-
-        print("loading " + type_path + " target.")
-
-        with open(os.path.join(data_dir, type_path + ".target"), "r") as f:
-            for text in f.readlines():  # each text is a line and a summary
-                tokenized = tokenizer.batch_encode_plus(
-                    [text], max_length=56, pad_to_max_length=True, return_tensors="pt"
-                )
-                self.target.append(tokenized)
-            f.close()
+        self.source = encode_file(tokenizer, os.path.join(data_dir, type_path + ".source"), max_source_length)
+        self.target = encode_file(tokenizer, os.path.join(data_dir, type_path + ".target"), max_target_length)

    def __len__(self):
        return len(self.source)
@@ -37,7 +37,20 @@ class SummarizationDataset(Dataset):
    def __getitem__(self, index):
        source_ids = self.source[index]["input_ids"].squeeze()
        target_ids = self.target[index]["input_ids"].squeeze()
-
-        src_mask = self.source[index]["attention_mask"].squeeze()  # might need to squeeze
-
+        src_mask = self.source[index]["attention_mask"].squeeze()
        return {"source_ids": source_ids, "source_mask": src_mask, "target_ids": target_ids}
+
+    @staticmethod
+    def trim_seq2seq_batch(batch, pad_token_id):
+        y = trim_batch(batch["target_ids"], pad_token_id)
+        source_ids, source_mask = trim_batch(batch["source_ids"], pad_token_id, attention_mask=batch["source_mask"])
+        return source_ids, source_mask, y
+
+    def collate_fn(self, batch):
+        input_ids = torch.stack([x["source_ids"] for x in batch])
+        masks = torch.stack([x["source_mask"] for x in batch])
+        target_ids = torch.stack([x["target_ids"] for x in batch])
+        pad_token_id = self.tokenizer.pad_token_id
+        y = trim_batch(target_ids, pad_token_id)
+        source_ids, source_mask = trim_batch(input_ids, pad_token_id, attention_mask=masks)
+        return {"source_ids": source_ids, "source_mask": source_mask, "target_ids": y}
@@ -15,7 +15,7 @@ wc -l cnn_articles_input_data.txt # should print 11490
 wc -l cnn_articles_reference_summaries.txt # should print 11490
 ```

-### Usage
+### Generating Summaries

 To create summaries for each article in dataset, run:
 ```bash
@@ -23,3 +23,7 @@ python evaluate_cnn.py cnn_articles_input_data.txt cnn_generated_articles_summar
 ```
 The default batch size, 8, fits in 16GB GPU memory, but may need to be adjusted to fit your system.
 The rouge scores "rouge1, rouge2, rougeL" are automatically created and saved in ``rouge_score.txt``.
+
+
+### Finetuning
+Pass model_type=t5 and model `examples/summarization/bart/finetune.py`
@@ -29,7 +29,14 @@ class TestT5Examples(unittest.TestCase):
        output_file_name = Path(tempfile.gettempdir()) / "utest_output_t5_sum.hypo"
        score_file_name = Path(tempfile.gettempdir()) / "utest_score_t5_sum.hypo"

-        testargs = ["evaluate_cnn.py", "t5-small", str(tmp), str(output_file_name), str(tmp), str(score_file_name)]
+        testargs = [
+            "evaluate_cnn.py",
+            "patrickvonplaten/t5-tiny-random",
+            str(tmp),
+            str(output_file_name),
+            str(tmp),
+            str(score_file_name),
+        ]

        with patch.object(sys, "argv", testargs):
            run_generate()
@@ -1,3 +1,4 @@
+import argparse
 import logging
 import os
 import random
@@ -38,7 +39,7 @@ MODEL_MODES = {
 }


-def set_seed(args):
+def set_seed(args: argparse.Namespace):
    random.seed(args.seed)
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
@@ -47,27 +48,29 @@ def set_seed(args):


 class BaseTransformer(pl.LightningModule):
-    def __init__(self, hparams, num_labels=None, mode="base"):
+    def __init__(self, hparams: argparse.Namespace, num_labels=None, mode="base", **config_kwargs):
        "Initialize a model."

        super(BaseTransformer, self).__init__()
        self.hparams = hparams
+        cache_dir = self.hparams.cache_dir if self.hparams.cache_dir else None
        self.hparams.model_type = self.hparams.model_type.lower()
        config = AutoConfig.from_pretrained(
            self.hparams.config_name if self.hparams.config_name else self.hparams.model_name_or_path,
            **({"num_labels": num_labels} if num_labels is not None else {}),
-            cache_dir=self.hparams.cache_dir if self.hparams.cache_dir else None,
+            cache_dir=cache_dir,
+            **config_kwargs,
        )
        tokenizer = AutoTokenizer.from_pretrained(
            self.hparams.tokenizer_name if self.hparams.tokenizer_name else self.hparams.model_name_or_path,
            do_lower_case=self.hparams.do_lower_case,
-            cache_dir=self.hparams.cache_dir if self.hparams.cache_dir else None,
+            cache_dir=cache_dir,
        )
        model = MODEL_MODES[mode].from_pretrained(
            self.hparams.model_name_or_path,
            from_tf=bool(".ckpt" in self.hparams.model_name_or_path),
            config=config,
-            cache_dir=self.hparams.cache_dir if self.hparams.cache_dir else None,
+            cache_dir=cache_dir,
        )
        self.config, self.tokenizer, self.model = config, tokenizer, model

@@ -102,8 +105,8 @@ class BaseTransformer(pl.LightningModule):
        self.lr_scheduler.step()

    def get_tqdm_dict(self):
-        tqdm_dict = {"loss": "{:.3f}".format(self.trainer.avg_loss), "lr": self.lr_scheduler.get_last_lr()[-1]}
-
+        avg_loss = getattr(self.trainer, "avg_loss", 0.0)
+        tqdm_dict = {"loss": "{:.3f}".format(avg_loss), "lr": self.lr_scheduler.get_last_lr()[-1]}
        return tqdm_dict

    def test_step(self, batch, batch_nb):
@@ -190,7 +193,7 @@ class BaseTransformer(pl.LightningModule):


 class LoggingCallback(pl.Callback):
-    def on_validation_end(self, trainer, pl_module):
+    def on_validation_end(self, trainer: pl.Trainer, pl_module: pl.LightningModule):
        logger.info("***** Validation results *****")
        if pl_module.is_logger():
            metrics = trainer.callback_metrics
@@ -199,7 +202,7 @@ class LoggingCallback(pl.Callback):
                if key not in ["log", "progress_bar"]:
                    logger.info("{} = {}\n".format(key, str(metrics[key])))

-    def on_test_end(self, trainer, pl_module):
+    def on_test_end(self, trainer: pl.Trainer, pl_module: pl.LightningModule):
        logger.info("***** Test results *****")

        if pl_module.is_logger():
@@ -254,7 +257,7 @@ def add_generic_args(parser, root_dir):
    parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")


-def generic_train(model, args):
+def generic_train(model: BaseTransformer, args: argparse.Namespace):
    # init model
    set_seed(args)

@@ -9,17 +9,17 @@ evaluated on the WMT English-German dataset.

 To be able to reproduce the authors' results on WMT English to German, you first need to download 
 the WMT14 en-de news datasets.
-Go on Stanford's official NLP [website](https://nlp.stanford.edu/projects/nmt/) and find "newstest2013.en" and "newstest2013.de" under WMT'14 English-German data or download the dataset directly via:
+Go on Stanford's official NLP [website](https://nlp.stanford.edu/projects/nmt/) and find "newstest2014.en" and "newstest2014.de" under WMT'14 English-German data or download the dataset directly via:

 ```bash
-curl https://nlp.stanford.edu/projects/nmt/data/wmt14.en-de/newstest2013.en > newstest2013.en
-curl https://nlp.stanford.edu/projects/nmt/data/wmt14.en-de/newstest2013.de > newstest2013.de
+curl https://nlp.stanford.edu/projects/nmt/data/wmt14.en-de/newstest2014.en > newstest2014.en
+curl https://nlp.stanford.edu/projects/nmt/data/wmt14.en-de/newstest2014.de > newstest2014.de
 ```

-You should have 3000 sentence in each file. You can verify this by running:
+You should have 2737 sentences in each file. You can verify this by running:

 ```bash
-wc -l newstest2013.en  # should give 3000
+wc -l newstest2014.en  # should give 2737
 ```

 ### Usage
@@ -29,8 +29,8 @@ Let's check the longest and shortest sentence in our file to find reasonable dec
 Get the longest and shortest sentence:

 ```bash 
-awk '{print NF}' newstest2013.en | sort -n | head -1 # shortest sentence has 1 word
-awk '{print NF}' newstest2013.en | sort -n | tail -1 # longest sentence has 106 words
+awk '{print NF}' newstest2014.en | sort -n | head -1 # shortest sentence has 2 word
+awk '{print NF}' newstest2014.en | sort -n | tail -1 # longest sentence has 91 words
 ```

 We will set our `max_length` to ~3 times the longest sentence and leave `min_length` to its default value of 0.
@@ -38,7 +38,7 @@ We decode with beam search `num_beams=4` as proposed in the paper. Also as is co

 To create translation for each in dataset and get a final BLEU score, run:
 ```bash
-python evaluate_wmt.py <path_to_newstest2013.en> newstest2013_de_translations.txt <path_to_newstest2013.de> newsstest2013_en_de_bleu.txt
+python evaluate_wmt.py <path_to_newstest2014.en> newstest2014_de_translations.txt <path_to_newstest2014.de> newsstest2014_en_de_bleu.txt
 ```
 the default batch size, 16, fits in 16GB GPU memory, but may need to be adjusted to fit your system.

@@ -15,8 +15,6 @@ def chunks(lst, n):


 def generate_translations(lns, output_file_path, model_size, batch_size, device):
-    output_file = Path(output_file_path).open("w")
-
    model = T5ForConditionalGeneration.from_pretrained(model_size)
    model.to(device)

@@ -27,27 +25,29 @@ def generate_translations(lns, output_file_path, model_size, batch_size, device)
    if task_specific_params is not None:
        model.config.update(task_specific_params.get("translation_en_to_de", {}))

-    for batch in tqdm(list(chunks(lns, batch_size))):
-        batch = [model.config.prefix + text for text in batch]
+    with Path(output_file_path).open("w") as output_file:
+        for batch in tqdm(list(chunks(lns, batch_size))):
+            batch = [model.config.prefix + text for text in batch]

-        dct = tokenizer.batch_encode_plus(batch, max_length=512, return_tensors="pt", pad_to_max_length=True)
+            dct = tokenizer.batch_encode_plus(batch, max_length=512, return_tensors="pt", pad_to_max_length=True)

-        input_ids = dct["input_ids"].to(device)
-        attention_mask = dct["attention_mask"].to(device)
+            input_ids = dct["input_ids"].to(device)
+            attention_mask = dct["attention_mask"].to(device)

-        translations = model.generate(input_ids=input_ids, attention_mask=attention_mask)
-        dec = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in translations]
+            translations = model.generate(input_ids=input_ids, attention_mask=attention_mask)
+            dec = [
+                tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in translations
+            ]

-        for hypothesis in dec:
-            output_file.write(hypothesis + "\n")
-            output_file.flush()
+            for hypothesis in dec:
+                output_file.write(hypothesis + "\n")


 def calculate_bleu_score(output_lns, refs_lns, score_path):
    bleu = corpus_bleu(output_lns, [refs_lns])
    result = "BLEU score: {}".format(bleu.score)
-    score_file = Path(score_path).open("w")
-    score_file.write(result)
+    with Path(score_path).open("w") as score_file:
+        score_file.write(result)


 def run_generate():
@@ -59,13 +59,13 @@ def run_generate():
        default="t5-base",
    )
    parser.add_argument(
-        "input_path", type=str, help="like wmt/newstest2013.en",
+        "input_path", type=str, help="like wmt/newstest2014.en",
    )
    parser.add_argument(
        "output_path", type=str, help="where to save translation",
    )
    parser.add_argument(
-        "reference_path", type=str, help="like wmt/newstest2013.de",
+        "reference_path", type=str, help="like wmt/newstest2014.de",
    )
    parser.add_argument(
        "score_path", type=str, help="where to save the bleu score",
@@ -82,12 +82,19 @@ def run_generate():

    dash_pattern = (" ##AT##-##AT## ", "-")

-    input_lns = [x.strip().replace(dash_pattern[0], dash_pattern[1]) for x in open(args.input_path).readlines()]
+    # Read input lines into python
+    with open(args.input_path, "r") as input_file:
+        input_lns = [x.strip().replace(dash_pattern[0], dash_pattern[1]) for x in input_file.readlines()]

    generate_translations(input_lns, args.output_path, args.model_size, args.batch_size, args.device)

-    output_lns = [x.strip() for x in open(args.output_path).readlines()]
-    refs_lns = [x.strip().replace(dash_pattern[0], dash_pattern[1]) for x in open(args.reference_path).readlines()]
+    # Read generated lines into python
+    with open(args.output_path, "r") as output_file:
+        output_lns = [x.strip() for x in output_file.readlines()]
+
+    # Read reference lines into python
+    with open(args.reference_path, "r") as reference_file:
+        refs_lns = [x.strip().replace(dash_pattern[0], dash_pattern[1]) for x in reference_file.readlines()]

    calculate_bleu_score(output_lns, refs_lns, args.score_path)

@@ -37,7 +37,7 @@ class TestT5Examples(unittest.TestCase):

        testargs = [
            "evaluate_wmt.py",
-            "t5-small",
+            "patrickvonplaten/t5-tiny-random",
            str(tmp_source),
            str(output_file_name),
            str(tmp_target),
@@ -0,0 +1,54 @@
+# MS-bert
+
+## Introduction
+
+This repository provides codes and models of MS-BERT.
+MS-BERT was pre-trained on notes from neurological examination for Multiple Sclerosis (MS) patients at St. Michael's Hospital in Toronto, Canada.
+
+## Data
+
+The dataset contained approximately 75,000 clinical notes, for about 5000 patients, totaling to over 35.7 million words.
+These notes were collected from patients who visited St. Michael's Hospital MS Clinic between 2015 to 2019.
+The notes contained a variety of information pertaining to a neurological exam.
+For example, a note can contain information on the patient's condition, their progress over time and diagnosis.
+The gender split within the dataset was observed to be 72% female and 28% male ([which reflects the natural discrepancy seen in MS][1]).
+Further sections will describe how MS-BERT was pre trained through the use of these clinically relevant and rich neurological notes.
+
+## Data pre-processing
+
+The data was pre-processed to remove any identifying information. This includes information on: patient names, doctor names, hospital names, patient identification numbers, phone numbers, addresses, and time. In order to de-identify the information, we used a curated database that contained patient and doctor information. This curated database was paired with regular expressions to find and remove any identifying pieces of information. Each of these identifiers were replaced with a specific token. These tokens were chosen based on three criteria: (1) they belong to the current BERT vocab, (2), they have relatively the same semantic meaning as the word they are replacing, and (3), the token is not found in the original unprocessed dataset. The replacements that met the criteria above were as follows: 
+
+Female first names -> Lucie
+
+Male first names -> Ezekiel
+
+Last/family names -> Salamanca.
+
+Dates -> 2010s
+
+Patient IDs -> 999
+
+Phone numbers -> 1718
+
+Addresses -> Silesia
+
+Time -> 1610
+
+Locations/Hospital/Clinic names -> Troy
+
+## Pre-training
+
+The starting point for our model is the already pre-trained and fine-tuned BLUE-BERT base. We further pre-train it using the masked language modelling task from the huggingface transformers [library](https://github.com/huggingface). 
+
+The hyperparameters can be found in the config file in this repository or [here](https://s3.amazonaws.com/models.huggingface.co/bert/NLP4H/ms_bert/config.json)
+
+## Acknowledgements
+
+We would like to thank the researchers and staff at the Data Science and Advanced Analytics (DSAA) team, St. Michael’s hospital, for providing consistent support and guidance throughout this project.
+We would also like to thank Dr. Marzyeh Ghassemi, Taylor Killan, Nathan Ng and Haoran Zhang for providing us the opportunity to work on this exciting project.
+
+## Disclaimer
+
+MS-BERT shows the results of research conducted at the Data Science and Advanced Analytics (DSAA) team, St. Michael’s hospital. The results produced by MS-BERT are not intended for direct diagnostic use or medical decision-making without review and oversight by a clinical professional. Individuals should not make decisions about their health solely on the basis of the results produced by MS-BERT. St. Michael’s hospital does not independently verify the validity or utility of the results produced by MS-BERT. If you have questions about the results produced by MS-BERT please consult a healthcare professional. More information about St. Michael’s hospital’s disclaimer policy can be found <here>. 
+
+[1]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3707353/
@@ -64,27 +64,8 @@ TensorFlow: 2.1.0
 Python: 3.7.6
 ```

-### Inferencing / prediction works with the current Transformers v2.4.1
-
-### Access this albert_xxlargev1_sqd2_512 fine-tuned model with "tried & true" code:
+### Access this albert_xxlargev1_sqd2_512 fine-tuned model with:

 ```python
-config_class, model_class, tokenizer_class = \
-        AlbertConfig, AlbertForQuestionAnswering, AlbertTokenizer
-
-model_name_or_path = "ahotrod/albert_xxlargev1_squad2_512"
-config = config_class.from_pretrained(model_name_or_path)
-tokenizer = tokenizer_class.from_pretrained(model_name_or_path, do_lower_case=True)
-model = model_class.from_pretrained(model_name_or_path, config=config)
-```
-
-### or the AutoModels (AutoConfig, AutoTokenizer & AutoModel) should also work, however I have yet to use them in my app & confirm:
-
-```python
-from transformers import AutoConfig, AutoTokenizer, AutoModel
-
-model_name_or_path = "ahotrod/albert_xxlargev1_squad2_512"
-config = AutoConfig.from_pretrained(model_name_or_path)
-tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, do_lower_case=True)
-model = AutoModel.from_pretrained(model_name_or_path, config=config)
-```
+tokenizer = AutoTokenizer.from_pretrained("ahotrod/albert_xxlargev1_squad2_512")
+model = AutoModelForQuestionAnswering.from_pretrained("ahotrod/albert_xxlargev1_squad2_512")
@@ -0,0 +1,56 @@
+## RoBERTa-large language model fine-tuned on SQuAD2.0
+
+### with the following results:
+
+```
+  "exact": 84.46896319380106,
+  "f1": 87.85388093408943,
+  "total": 11873,
+  "HasAns_exact": 81.37651821862349,
+  "HasAns_f1": 88.1560607844881,
+  "HasAns_total": 5928,
+  "NoAns_exact": 87.55256518082422,
+  "NoAns_f1": 87.55256518082422,
+  "NoAns_total": 5945,
+  "best_exact": 84.46896319380106,
+  "best_exact_thresh": 0.0,
+  "best_f1": 87.85388093408929,
+  "best_f1_thresh": 0.0
+```
+### from script:
+```
+python ${EXAMPLES}/run_squad.py \
+  --model_type roberta \
+  --model_name_or_path roberta-large \
+  --do_train \
+  --do_eval \
+  --train_file ${SQUAD}/train-v2.0.json \
+  --predict_file ${SQUAD}/dev-v2.0.json \
+  --version_2_with_negative \
+  --do_lower_case \
+  --num_train_epochs 3 \
+  --warmup_steps 1642 \
+  --weight_decay 0.01 \
+  --learning_rate 3e-5 \
+  --adam_epsilon 1e-6 \
+  --max_seq_length 512 \
+  --doc_stride 128 \
+  --per_gpu_train_batch_size 8 \
+  --gradient_accumulation_steps 6 \
+  --per_gpu_eval_batch_size 48 \
+  --threads 12 \
+  --logging_steps 50 \
+  --save_steps 2000 \
+  --overwrite_output_dir \
+  --output_dir ${MODEL_PATH}
+$@
+```
+### using the following system & software:
+```
+Transformers: 2.7.0
+PyTorch: 1.4.0
+TensorFlow: 2.1.0
+Python: 3.7.7
+OS/Platform: Linux-5.3.0-46-generic-x86_64-with-debian-buster-sid
+CPU/GPU: Intel i9-9900K / NVIDIA Titan RTX 24GB
+```
@@ -56,22 +56,8 @@ PyTorch: 1.4.0
 TensorFlow: 2.1.0
 Python: 3.7.6
 ```
-### Inferencing / prediction works with Transformers v2.4.1, the latest version tested
-
 ### Utilize this xlnet_large_squad2_512 fine-tuned model with:
 ```python
-config_class, model_class, tokenizer_class = \
-        XLNetConfig, XLNetforQuestionAnswering, XLNetTokenizer
-model_name_or_path = "ahotrod/xlnet_large_squad2_512"
-config = config_class.from_pretrained(model_name_or_path)
-tokenizer = tokenizer_class.from_pretrained(model_name_or_path, do_lower_case=True)
-model = model_class.from_pretrained(model_name_or_path, config=config)
-```
-### or the AutoModels (AutoConfig, AutoTokenizer & AutoModel) should also work, however I have yet to use them in my apps & confirm:
-```python
-from transformers import AutoConfig, AutoTokenizer, AutoModel
-model_name_or_path = "ahotrod/xlnet_large_squad2_512"
-config = AutoConfig.from_pretrained(model_name_or_path)
-tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, do_lower_case=True)
-model = AutoModel.from_pretrained(model_name_or_path, config=config)
+tokenizer = AutoTokenizer.from_pretrained("ahotrod/xlnet_large_squad2_512")
+model = AutoModelForQuestionAnswering.from_pretrained("ahotrod/xlnet_large_squad2_512")
 ```
@@ -0,0 +1,8 @@
+---
+tags:
+- exbert
+---
+
+<a href="https://huggingface.co/exbert/?model=albert-base-v1">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
@@ -0,0 +1,8 @@
+---
+tags:
+- exbert
+---
+
+<a href="https://huggingface.co/exbert/?model=albert-xxlarge-v2">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
@@ -0,0 +1,38 @@
+---
+thumbnail: https://huggingface.co/front/thumbnails/allenai.png
+---
+
+# BioMed-RoBERTa-base
+
+BioMed-RoBERTa-base is a language model based on the RoBERTa-base (Liu et. al, 2019) architecture. We adapt RoBERTa-base to 2.68 million scientific papers from the [Semantic Scholar](https://www.semanticscholar.org) corpus via continued pretraining. This amounts to 7.55B tokens and 47GB of data. We use the full text of the papers in training, not just abstracts.
+
+Specific details of the adaptive pretraining procedure can be found in Gururangan et. al, 2020. 
+
+
+## Evaluation
+
+BioMed-RoBERTa achieves competitive performance to state of the art models on a number of NLP tasks in the biomedical domain (numbers are mean (standard deviation) over 3+ random seeds)
+
+
+| Task         | Task Type           | RoBERTa-base | BioMed-RoBERTa-base |
+|--------------|---------------------|--------------|---------------------|
+| RCT-180K     | Text Classification | 86.4 (0.3)   | 86.9 (0.2)          |
+| ChemProt     | Relation Extraction | 81.1 (1.1)   | 83.0 (0.7)          |
+| JNLPBA       | NER                 | 74.3 (0.2)   | 75.2 (0.1)          |
+| BC5CDR       | NER                 | 85.6 (0.1)   | 87.8 (0.1)          |
+| NCBI-Disease | NER                 | 86.6 (0.3)   | 87.1 (0.8)          |
+
+More evaluations TBD.
+
+## Citation
+
+If using this model, please cite the following paper:
+
+```bibtex
+@inproceedings{domains,
+ author = {Suchin Gururangan and Ana Marasović and Swabha Swayamdipta and Kyle Lo and Iz Beltagy and Doug Downey and Noah A. Smith},
+ title = {Don't Stop Pretraining: Adapt Language Models to Domains and Tasks},
+ year = {2020},
+ booktitle = {Proceedings of ACL},
+}
+```
@@ -0,0 +1,5 @@
+---
+tags:
+- summarization
+---
+
@@ -0,0 +1,5 @@
+---
+tags:
+- summarization
+---
+
@@ -0,0 +1,8 @@
+---
+tags:
+- exbert
+---
+
+<a href="https://huggingface.co/exbert/?model=bert-base-cased">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
@@ -1,8 +1,14 @@
 ---
 language: german
 thumbnail: https://static.tildacdn.com/tild6438-3730-4164-b266-613634323466/german_bert.png
+tags:
+- exbert
 ---

+<a href="https://huggingface.co/exbert/?model=bert-base-german-cased">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
+
 # German BERT
 ![bert_image](https://static.tildacdn.com/tild6438-3730-4164-b266-613634323466/german_bert.png)
 ## Overview
@@ -18,6 +24,7 @@ thumbnail: https://static.tildacdn.com/tild6438-3730-4164-b266-613634323466/germ
 - We trained 810k steps with a batch size of 1024 for sequence length 128 and 30k steps with sequence length 512. Training took about 9 days.
 - As training data we used the latest German Wikipedia dump (6GB of raw txt files), the OpenLegalData dump (2.4 GB) and news articles (3.6 GB).
 - We cleaned the data dumps with tailored scripts and segmented sentences with spacy v2.1. To create tensorflow records we used the recommended sentencepiece library for creating the word piece vocabulary and tensorflow scripts to convert the text to data usable by BERT.
+- Update April 3rd, 2020: updated the vocab file on deepset s3 to adjust tokenization of punctuation.

 See https://deepset.ai/german-bert for more details

@@ -0,0 +1,8 @@
+---
+tags:
+- exbert
+---
+
+<a href="https://huggingface.co/exbert/?model=bert-base-uncased">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
@@ -1,9 +1,111 @@
-# CamemBERT 
+---
+language: french
+---

-CamemBERT is a state-of-the-art language model for French based on the RoBERTa architecture pretrained on the French subcorpus of the newly available multilingual corpus OSCAR.  
+# CamemBERT: a Tasty French Language Model

-CamemBERT was originally evaluated on four different downstream tasks for French: part-of-speech (POS) tagging, dependency parsing, named entity recognition (NER) and natural language inference (NLI); improving the state of the art for most tasks over previous monolingual and multilingual approaches, which confirms the effectiveness of large pretrained language models for French.   
+## Introduction

-CamemBERT was trained and evaluated by Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.  
+[CamemBERT](https://arxiv.org/abs/1911.03894) is a state-of-the-art language model for French based on the RoBERTa architecture. 
+
+It is now available on Hugging Face in 6 different versions varying the number of parameters, the amount of pretraining data and the pretraining data source domains. 
+
+For further information or requests, please go to [Camembert Website](https://camembert-model.fr/)
+
+## Pre-trained models
+
+| Model                          | #params                        | Arch. | Training data                     |
+|--------------------------------|--------------------------------|-------|-----------------------------------|
+| `camembert-base` | 110M   | Base  | OSCAR (138 GB of text)            |
+| `camembert` / `camembert-large`              | 335M    | Large | CCNet (135 GB of text)            |
+| `camembert` / `camembert-base-ccnet`         | 110M    | Base  | CCNet (135 GB of text)            |
+| `camembert` / `camembert-base-wikipedia-4gb` | 110M    | Base  | Wikipedia (4 GB of text)          |
+| `camembert` / `camembert-base-oscar-4gb`     | 110M    | Base  | Subsample of OSCAR (4 GB of text) |
+| `camembert` / `camembert-base-ccnet-4gb`     | 110M    | Base  | Subsample of CCNet (4 GB of text) |
+
+## How to use CamemBERT with HuggingFace
+
+##### Load CamemBERT and its sub-word tokenizer :
+```python
+from transformers import CamembertModel, CamembertTokenizer
+
+tokenizer = CamembertTokenizer.from_pretrained("camembert-base")
+camembert = CamembertModel.from_pretrained("camembert-base")
+
+camembert.eval()  # disable dropout (or leave in train mode to finetune)
+
+```
+
+##### Filling masks using pipeline 
+```python
+from transformers import pipeline 
+
+camembert_fill_mask  = pipeline("fill-mask",model="camembert-base",tokenizer="camembert-base")
+results = camembert_fill_mask("Le camembert est <mask> :)")
+# results
+#[{'sequence': '<s> Le camembert est délicieux :)</s>', 'score': 0.4909103214740753, 'token': 7200},
+# {'sequence': '<s> Le camembert est excellent :)</s>', 'score': 0.10556930303573608, 'token': 2183}, 
+# {'sequence': '<s> Le camembert est succulent :)</s>', 'score': 0.03453315049409866, 'token': 26202}, 
+# {'sequence': '<s> Le camembert est meilleur :)</s>', 'score': 0.03303130343556404, 'token': 528}, 
+# {'sequence': '<s> Le camembert est parfait :)</s>', 'score': 0.030076518654823303, 'token': 1654}]
+
+```
+
+##### Extract contextual embedding features from Camembert output 
+```python
+import torch
+# Tokenize in sub-words with SentencePiece
+tokenized_sentence = tokenizer.tokenize("J'aime le camembert !")
+# ['▁J', "'", 'aime', '▁le', '▁ca', 'member', 't', '▁!'] 
+
+# 1-hot encode and add special starting and end tokens 
+encoded_sentence = tokenizer.encode(tokenized_sentence)
+# [5, 121, 11, 660, 16, 730, 25543, 110, 83, 6] 
+# NB: can do in one step : tokenize.encode("J'aime le camembert !")
+
+# Feed to Camembert as a torch tensor (batch dim 1)
+encoded_sentence = torch.tensor(encoded_sentence).unsqueeze(0)
+embeddings, _ = camembert(encoded_sentence)
+# embeddings.detach()
+# embeddings.size torch.Size([1, 10, 768])
+# tensor([[[-0.0254,  0.0235,  0.1027,  ..., -0.1459, -0.0205, -0.0116],
+#         [ 0.0606, -0.1811, -0.0418,  ..., -0.1815,  0.0880, -0.0766],
+#         [-0.1561, -0.1127,  0.2687,  ..., -0.0648,  0.0249,  0.0446],
+#         ...,
+```
+
+##### Extract contextual embedding features from all Camembert layers
+```python
+from transformers import CamembertConfig
+# (Need to reload the model with new config)
+config = CamembertConfig.from_pretrained("camembert-base", output_hidden_states=True)
+camembert = CamembertModel.from_pretrained("camembert-base",config=config)
+
+embeddings, _, all_layer_embeddings = camembert(encoded_sentence)
+#  all_layer_embeddings list of len(all_layer_embeddings) == 13 (input embedding layer + 12 self attention layers)
+all_layer_embeddings[5]
+# layer 5 contextual embedding : size torch.Size([1, 10, 768])
+#tensor([[[-0.0032,  0.0075,  0.0040,  ..., -0.0025, -0.0178, -0.0210],
+#         [-0.0996, -0.1474,  0.1057,  ..., -0.0278,  0.1690, -0.2982],
+#         [ 0.0557, -0.0588,  0.0547,  ..., -0.0726, -0.0867,  0.0699],
+#         ...,
+```
+
+
+## Authors 
+
+CamemBERT was trained and evaluated by Louis Martin\*, Benjamin Muller\*, Pedro Javier Ortiz Suárez\*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.
+
+
+## Citation
+If you use our work, please cite:
+
+```bibtex
+@inproceedings{martin2020camembert,
+  title={CamemBERT: a Tasty French Language Model},
+  author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
+  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
+  year={2020}
+}
+```

-Preprint can be found [CamemBERT: a Tasty French Language Model](https://arxiv.org/abs/1911.03894)  
@@ -0,0 +1,8 @@
+---
+tags:
+- exbert
+---
+
+<a href="https://huggingface.co/exbert/?model=distilbert-base-uncased">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
@@ -0,0 +1,8 @@
+---
+tags:
+- exbert
+---
+
+<a href="https://huggingface.co/exbert/?model=distilgpt2">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
@@ -0,0 +1,8 @@
+---
+tags:
+- exbert
+---
+
+<a href="https://huggingface.co/exbert/?model=distilroberta-base">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
@@ -1,3 +1,8 @@
+---
+tags:
+- exbert
+---
+
 ## CS224n SQuAD2.0 Project Dataset
 The goal of this model is to save CS224n students GPU time when establising
 baselines to beat for the [Default Final Project](http://web.stanford.edu/class/cs224n/project/default-final-project-handout.pdf).
@@ -11,6 +16,10 @@ set, students must make sure not to use the official SQuAD2.0 dev set in any way
 — including the use of models fine-tuned on the official SQuAD2.0, since they
 used the official SQuAD2.0 dev set for model selection.

+<a href="https://huggingface.co/exbert/?model=elgeish/cs224n-squad2.0-albert-base-v2">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
+
 ## Results
 ```json
 {
@@ -67,6 +76,18 @@ used the official SQuAD2.0 dev set for model selection.
 }
 ```

+## How to Cite
+```BibTeX
+@misc{elgeish2020gestalt,
+  title={Gestalt: a Stacking Ensemble for SQuAD2.0},
+  author={Mohamed El-Geish},
+  journal={arXiv e-prints},
+  archivePrefix={arXiv},
+  eprint={2004.07067},
+  year={2020},
+}
+```
+
 ## Related Models
 * [elgeish/cs224n-squad2.0-albert-large-v2](https://huggingface.co/elgeish/cs224n-squad2.0-albert-large-v2)
 * [elgeish/cs224n-squad2.0-albert-xxlarge-v1](https://huggingface.co/elgeish/cs224n-squad2.0-albert-xxlarge-v1)
@@ -1,3 +1,8 @@
+---
+tags:
+- exbert
+---
+
 ## CS224n SQuAD2.0 Project Dataset
 The goal of this model is to save CS224n students GPU time when establising
 baselines to beat for the [Default Final Project](http://web.stanford.edu/class/cs224n/project/default-final-project-handout.pdf).
@@ -11,6 +16,10 @@ set, students must make sure not to use the official SQuAD2.0 dev set in any way
 — including the use of models fine-tuned on the official SQuAD2.0, since they
 used the official SQuAD2.0 dev set for model selection.

+<a href="https://huggingface.co/exbert/?model=elgeish/cs224n-squad2.0-albert-large-v2">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
+
 ## Results
 ```json
 {
@@ -67,6 +76,18 @@ used the official SQuAD2.0 dev set for model selection.
 }
 ```

+## How to Cite
+```BibTeX
+@misc{elgeish2020gestalt,
+  title={Gestalt: a Stacking Ensemble for SQuAD2.0},
+  author={Mohamed El-Geish},
+  journal={arXiv e-prints},
+  archivePrefix={arXiv},
+  eprint={2004.07067},
+  year={2020},
+}
+```
+
 ## Related Models
 * [elgeish/cs224n-squad2.0-albert-base-v2](https://huggingface.co/elgeish/cs224n-squad2.0-albert-base-v2)
 * [elgeish/cs224n-squad2.0-albert-xxlarge-v1](https://huggingface.co/elgeish/cs224n-squad2.0-albert-xxlarge-v1)
@@ -1,3 +1,8 @@
+---
+tags:
+- exbert
+---
+
 ## CS224n SQuAD2.0 Project Dataset
 The goal of this model is to save CS224n students GPU time when establising
 baselines to beat for the [Default Final Project](http://web.stanford.edu/class/cs224n/project/default-final-project-handout.pdf).
@@ -11,6 +16,10 @@ set, students must make sure not to use the official SQuAD2.0 dev set in any way
 — including the use of models fine-tuned on the official SQuAD2.0, since they
 used the official SQuAD2.0 dev set for model selection.

+<a href="https://huggingface.co/exbert/?model=elgeish/cs224n-squad2.0-albert-xxlarge-v1">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
+
 ## Results
 ```json
 {
@@ -67,6 +76,18 @@ used the official SQuAD2.0 dev set for model selection.
 }
 ```

+## How to Cite
+```BibTeX
+@misc{elgeish2020gestalt,
+  title={Gestalt: a Stacking Ensemble for SQuAD2.0},
+  author={Mohamed El-Geish},
+  journal={arXiv e-prints},
+  archivePrefix={arXiv},
+  eprint={2004.07067},
+  year={2020},
+}
+```
+
 ## Related Models
 * [elgeish/cs224n-squad2.0-albert-base-v2](https://huggingface.co/elgeish/cs224n-squad2.0-albert-base-v2)
 * [elgeish/cs224n-squad2.0-albert-large-v2](https://huggingface.co/elgeish/cs224n-squad2.0-albert-large-v2)
@@ -67,6 +67,18 @@ used the official SQuAD2.0 dev set for model selection.
 }
 ```

+## How to Cite
+```BibTeX
+@misc{elgeish2020gestalt,
+  title={Gestalt: a Stacking Ensemble for SQuAD2.0},
+  author={Mohamed El-Geish},
+  journal={arXiv e-prints},
+  archivePrefix={arXiv},
+  eprint={2004.07067},
+  year={2020},
+}
+```
+
 ## Related Models
 * [elgeish/cs224n-squad2.0-albert-base-v2](https://huggingface.co/elgeish/cs224n-squad2.0-albert-base-v2)
 * [elgeish/cs224n-squad2.0-albert-large-v2](https://huggingface.co/elgeish/cs224n-squad2.0-albert-large-v2)
@@ -67,6 +67,18 @@ used the official SQuAD2.0 dev set for model selection.
 }
 ```

+## How to Cite
+```BibTeX
+@misc{elgeish2020gestalt,
+  title={Gestalt: a Stacking Ensemble for SQuAD2.0},
+  author={Mohamed El-Geish},
+  journal={arXiv e-prints},
+  archivePrefix={arXiv},
+  eprint={2004.07067},
+  year={2020},
+}
+```
+
 ## Related Models
 * [elgeish/cs224n-squad2.0-albert-base-v2](https://huggingface.co/elgeish/cs224n-squad2.0-albert-base-v2)
 * [elgeish/cs224n-squad2.0-albert-large-v2](https://huggingface.co/elgeish/cs224n-squad2.0-albert-large-v2)
@@ -0,0 +1,34 @@
+---
+language: english
+thumbnail: https://huggingface.co/front/thumbnails/google.png
+---
+
+## ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
+
+**ELECTRA** is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) dataset.
+
+For a detailed description and experimental results, please refer to our paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://openreview.net/pdf?id=r1xMH1BtvB).
+
+This repository contains code to pre-train ELECTRA, including small ELECTRA models on a single GPU. It also supports fine-tuning ELECTRA on downstream tasks including classification tasks (e.g,. [GLUE](https://gluebenchmark.com/)), QA tasks (e.g., [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)), and sequence tagging tasks (e.g., [text chunking](https://www.clips.uantwerpen.be/conll2000/chunking/)).
+
+## How to use the discriminator in `transformers`
+
+```python
+from transformers import ElectraForPreTraining, ElectraTokenizerFast
+import torch
+
+discriminator = ElectraForPreTraining.from_pretrained("google/electra-base-discriminator")
+tokenizer = ElectraTokenizerFast.from_pretrained("google/electra-base-discriminator")
+
+sentence = "The quick brown fox jumps over the lazy dog"
+fake_sentence = "The quick brown fox fake over the lazy dog"
+
+fake_tokens = tokenizer.tokenize(fake_sentence)
+fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt")
+discriminator_outputs = discriminator(fake_inputs)
+predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2)
+
+[print("%7s" % token, end="") for token in fake_tokens]
+
+[print("%7s" % int(prediction), end="") for prediction in predictions.tolist()]
+```
@@ -0,0 +1,29 @@
+---
+language: english
+thumbnail: https://huggingface.co/front/thumbnails/google.png
+---
+
+## ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
+
+**ELECTRA** is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) dataset.
+
+For a detailed description and experimental results, please refer to our paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://openreview.net/pdf?id=r1xMH1BtvB).
+
+This repository contains code to pre-train ELECTRA, including small ELECTRA models on a single GPU. It also supports fine-tuning ELECTRA on downstream tasks including classification tasks (e.g,. [GLUE](https://gluebenchmark.com/)), QA tasks (e.g., [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)), and sequence tagging tasks (e.g., [text chunking](https://www.clips.uantwerpen.be/conll2000/chunking/)).
+
+## How to use the generator in `transformers`
+
+```python
+from transformers import pipeline
+
+fill_mask = pipeline(
+	"fill-mask",
+	model="google/electra-base-generator",
+	tokenizer="google/electra-base-generator"
+)
+
+print(
+	fill_mask(f"HuggingFace is creating a {nlp.tokenizer.mask_token} that the community uses to solve NLP tasks.")
+)
+
+```
@@ -0,0 +1,34 @@
+---
+language: english
+thumbnail: https://huggingface.co/front/thumbnails/google.png
+---
+
+## ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
+
+**ELECTRA** is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) dataset.
+
+For a detailed description and experimental results, please refer to our paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://openreview.net/pdf?id=r1xMH1BtvB).
+
+This repository contains code to pre-train ELECTRA, including small ELECTRA models on a single GPU. It also supports fine-tuning ELECTRA on downstream tasks including classification tasks (e.g,. [GLUE](https://gluebenchmark.com/)), QA tasks (e.g., [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)), and sequence tagging tasks (e.g., [text chunking](https://www.clips.uantwerpen.be/conll2000/chunking/)).
+
+## How to use the discriminator in `transformers`
+
+```python
+from transformers import ElectraForPreTraining, ElectraTokenizerFast
+import torch
+
+discriminator = ElectraForPreTraining.from_pretrained("google/electra-large-discriminator")
+tokenizer = ElectraTokenizerFast.from_pretrained("google/electra-large-discriminator")
+
+sentence = "The quick brown fox jumps over the lazy dog"
+fake_sentence = "The quick brown fox fake over the lazy dog"
+
+fake_tokens = tokenizer.tokenize(fake_sentence)
+fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt")
+discriminator_outputs = discriminator(fake_inputs)
+predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2)
+
+[print("%7s" % token, end="") for token in fake_tokens]
+
+[print("%7s" % int(prediction), end="") for prediction in predictions.tolist()]
+```
@@ -0,0 +1,29 @@
+---
+language: english
+thumbnail: https://huggingface.co/front/thumbnails/google.png
+---
+
+## ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
+
+**ELECTRA** is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) dataset.
+
+For a detailed description and experimental results, please refer to our paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://openreview.net/pdf?id=r1xMH1BtvB).
+
+This repository contains code to pre-train ELECTRA, including small ELECTRA models on a single GPU. It also supports fine-tuning ELECTRA on downstream tasks including classification tasks (e.g,. [GLUE](https://gluebenchmark.com/)), QA tasks (e.g., [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)), and sequence tagging tasks (e.g., [text chunking](https://www.clips.uantwerpen.be/conll2000/chunking/)).
+
+## How to use the generator in `transformers`
+
+```python
+from transformers import pipeline
+
+fill_mask = pipeline(
+	"fill-mask",
+	model="google/electra-large-generator",
+	tokenizer="google/electra-large-generator"
+)
+
+print(
+	fill_mask(f"HuggingFace is creating a {nlp.tokenizer.mask_token} that the community uses to solve NLP tasks.")
+)
+
+```
@@ -0,0 +1,34 @@
+---
+language: english
+thumbnail: https://huggingface.co/front/thumbnails/google.png
+---
+
+## ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
+
+**ELECTRA** is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) dataset.
+
+For a detailed description and experimental results, please refer to our paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://openreview.net/pdf?id=r1xMH1BtvB).
+
+This repository contains code to pre-train ELECTRA, including small ELECTRA models on a single GPU. It also supports fine-tuning ELECTRA on downstream tasks including classification tasks (e.g,. [GLUE](https://gluebenchmark.com/)), QA tasks (e.g., [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)), and sequence tagging tasks (e.g., [text chunking](https://www.clips.uantwerpen.be/conll2000/chunking/)).
+
+## How to use the discriminator in `transformers`
+
+```python
+from transformers import ElectraForPreTraining, ElectraTokenizerFast
+import torch
+
+discriminator = ElectraForPreTraining.from_pretrained("google/electra-small-discriminator")
+tokenizer = ElectraTokenizerFast.from_pretrained("google/electra-small-discriminator")
+
+sentence = "The quick brown fox jumps over the lazy dog"
+fake_sentence = "The quick brown fox fake over the lazy dog"
+
+fake_tokens = tokenizer.tokenize(fake_sentence)
+fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt")
+discriminator_outputs = discriminator(fake_inputs)
+predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2)
+
+[print("%7s" % token, end="") for token in fake_tokens]
+
+[print("%7s" % int(prediction), end="") for prediction in predictions.tolist()]
+```
@@ -0,0 +1,29 @@
+---
+language: english
+thumbnail: https://huggingface.co/front/thumbnails/google.png
+---
+
+## ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
+
+**ELECTRA** is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) dataset.
+
+For a detailed description and experimental results, please refer to our paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://openreview.net/pdf?id=r1xMH1BtvB).
+
+This repository contains code to pre-train ELECTRA, including small ELECTRA models on a single GPU. It also supports fine-tuning ELECTRA on downstream tasks including classification tasks (e.g,. [GLUE](https://gluebenchmark.com/)), QA tasks (e.g., [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)), and sequence tagging tasks (e.g., [text chunking](https://www.clips.uantwerpen.be/conll2000/chunking/)).
+
+## How to use the generator in `transformers`
+
+```python
+from transformers import pipeline
+
+fill_mask = pipeline(
+	"fill-mask",
+	model="google/electra-small-generator",
+	tokenizer="google/electra-small-generator"
+)
+
+print(
+	fill_mask(f"HuggingFace is creating a {nlp.tokenizer.mask_token} that the community uses to solve NLP tasks.")
+)
+
+```
@@ -0,0 +1,8 @@
+---
+tags:
+- exbert
+---
+
+<a href="https://huggingface.co/exbert/?model=gpt2">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
@@ -6,14 +6,14 @@ language: dutch

 This model is the multilingual model provided by the Google research team with a fine-tuned dutch Q&A downstream task.

-## Details of the language model(bert-base-multilingual-cased)
+## Details of the language model

 Language model ([**bert-base-multilingual-cased**](https://github.com/google-research/bert/blob/master/multilingual.md)):
 12-layer, 768-hidden, 12-heads, 110M parameters.
 Trained on cased text in the top 104 languages with the largest Wikipedias.

-## Details of the downstream task - Dataset
-Using the `mtranslate` Python module, [**SQuAD2.0**](https://rajpurkar.github.io/SQuAD-explorer/) was machine-translated. In order to find the start tokens the direct translations of the answers were searched in the corresponding paragraphs. Since the answer could not always be found in the text, due to the different translations depending on the context (missing context in the pure answer), a loss of question-answer examples occurred. This is a potential problem where errors can occur in the data set (but in the end it was a quick and dirty solution that worked well enough for my task).
+## Details of the downstream task
+Using the `mtranslate` Python module, [**SQuAD2.0**](https://rajpurkar.github.io/SQuAD-explorer/) was machine-translated. In order to find the start tokens, the direct translations of the answers were searched in the corresponding paragraphs. Due to the different translations depending on the context (missing context in the pure answer), the answer could not always be found in the text, and thus a loss of question-answer examples occurred. This is a potential problem where errors can occur in the data set.

 | Dataset                | # Q&A |
 | ---------------------- | ----- |
@@ -22,29 +22,73 @@ Using the `mtranslate` Python module, [**SQuAD2.0**](https://rajpurkar.github.io
 | SQuAD2.0 Dev           | 12  K |
 | Dutch SQuAD2.0 Dev     | 10  K |

+
+## Model benchmark
+
+
+| Model                | EM/F1 |HasAns (EM/F1) | NoAns |
+| ---------------------- | ----- | ----- | ----- |
+| [robBERT](https://huggingface.co/pdelobelle/robBERT-base)   | 58.04/60.95  | 33.08/40.64 | 73.67 |
+| [dutchBERT](https://huggingface.co/wietsedv/bert-base-dutch-cased)   | 64.25/68.45 | 45.59/56.49  | 75.94 |
+| [multiBERT](https://huggingface.co/bert-base-multilingual-cased) | **67.38**/**71.36**  | 47.42/57.76 | 79.88 |
+
 ## Model training

-The model was trained on a Tesla V100 GPU with the following command:
+The model was trained on a **Tesla V100** GPU with the following command:

 ```python
 export SQUAD_DIR=path/to/nl_squad

-python run_squad.py \
+python run_squad.py 
  --model_type bert \
  --model_name_or_path bert-base-multilingual-cased \
-  --version_2_with_negative \
  --do_train \
  --do_eval \
-  --train_file $SQUAD_DIR/train_nl-v2.0.json \
-  --predict_file $SQUAD_DIR/dev_nl-v2.0.json \
-  --per_gpu_train_batch_size 12 \
-  --learning_rate 3e-5 \
-  --num_train_epochs 2.0 \
+  --train_file $SQUAD_DIR/nl_squadv2_train_clean.json \
+  --predict_file $SQUAD_DIR/nl_squadv2_dev_clean.json \
+  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
-  --output_dir /tmp/output_dir/
+  --save_steps=8000 \
+  --output_dir ../../output \
+  --overwrite_cache \
+  --overwrite_output_dir
 ```

 **Results**:

-{'exact': **67.38**, 'f1': **71.36**} 
+{'exact': 67.38028751680629, 'f1': 71.362297054268, 'total': 9669, 'HasAns_exact': 47.422126745435015, 'HasAns_f1': 57.761023151910734, 'HasAns_total': 3724, 'NoAns_exact': 79.88225399495374, 'NoAns_f1': 79.88225399495374, 'NoAns_total': 5945, 'best_exact': 67.53542248422795, 'best_exact_thresh': 0.0, 'best_f1': 71.36229705426837, 'best_f1_thresh': 0.0}
+
+## Model in action
+
+Fast usage with **pipelines**:
+
+```python
+from transformers import pipeline
+
+qa_pipeline = pipeline(
+    "question-answering",
+    model="henryk/bert-base-multilingual-cased-finetuned-dutch-squad2",
+    tokenizer="henryk/bert-base-multilingual-cased-finetuned-dutch-squad2"
+)
+
+qa_pipeline({
+    'context': "Amsterdam is de hoofdstad en de dichtstbevolkte stad van Nederland.",
+    'question': "Wat is de hoofdstad van Nederland?"})
+
+```
+
+# Output:
+
+```json
+{
+  "score": 0.83,
+  "start": 0, 
+  "end": 9,
+  "answer": "Amsterdam"
+}
+```
+
+## Contact
+
+Please do not hesitate to contact me via [LinkedIn](https://www.linkedin.com/in/henryk-borzymowski-0755a2167/) if you want to discuss or get access to the Dutch version of SQuAD.
@@ -0,0 +1,94 @@
+---
+language: polish
+---
+
+# Multilingual + Polish SQuAD1.1
+
+This model is the multilingual model provided by the Google research team with a fine-tuned polish Q&A downstream task.
+
+## Details of the language model
+
+Language model ([**bert-base-multilingual-cased**](https://github.com/google-research/bert/blob/master/multilingual.md)):
+12-layer, 768-hidden, 12-heads, 110M parameters.
+Trained on cased text in the top 104 languages with the largest Wikipedias.
+
+## Details of the downstream task
+Using the `mtranslate` Python module, [**SQuAD1.1**](https://rajpurkar.github.io/SQuAD-explorer/) was machine-translated. In order to find the start tokens, the direct translations of the answers were searched in the corresponding paragraphs. Due to the different translations depending on the context (missing context in the pure answer), the answer could not always be found in the text, and thus a loss of question-answer examples occurred. This is a potential problem where errors can occur in the data set.
+
+| Dataset                | # Q&A |
+| ---------------------- | ----- |
+| SQuAD1.1 Train         | 87.7 K |
+| Polish SQuAD1.1 Train   | 39.5 K |
+| SQuAD1.1 Dev           |  10.6 K |
+| Polish SQuAD1.1 Dev     |  2.6 K |
+
+
+## Model benchmark
+
+| Model                | EM | F1 |
+| ---------------------- | ----- | ----- |
+| [SlavicBERT](https://huggingface.co/DeepPavlov/bert-base-bg-cs-pl-ru-cased)   | **60.89** | 71.68 |
+| [polBERT](https://huggingface.co/dkleczek/bert-base-polish-uncased-v1)   | 57.46 | 68.87 |
+| [multiBERT](https://huggingface.co/bert-base-multilingual-cased) | 60.67 | **71.89** |
+| [xlm](https://huggingface.co/xlm-mlm-100-1280)     | 47.98 | 59.42 |
+## Model training
+
+The model was trained on a **Tesla V100** GPU with the following command:
+
+```python
+export SQUAD_DIR=path/to/pl_squad
+
+python run_squad.py 
+  --model_type bert \
+  --model_name_or_path bert-base-multilingual-cased \
+  --do_train \
+  --do_eval \
+  --train_file $SQUAD_DIR/pl_squadv1_train_clean.json \
+  --predict_file $SQUAD_DIR/pl_squadv1_dev_clean.json \
+  --num_train_epochs 2 \
+  --max_seq_length 384 \
+  --doc_stride 128 \
+  --save_steps=8000 \
+  --output_dir ../../output \
+  --overwrite_cache \
+  --overwrite_output_dir
+```
+
+**Results**:
+
+{'exact': 60.670731707317074, 'f1': 71.8952193697293, 'total': 2624, 'HasAns_exact': 60.670731707317074, 'HasAns_f1': 71.8952193697293,
+'HasAns_total': 2624, 'best_exact': 60.670731707317074, 'best_exact_thresh': 0.0, 'best_f1': 71.8952193697293, 'best_f1_thresh': 0.0}
+
+## Model in action
+
+Fast usage with **pipelines**:
+
+```python
+from transformers import pipeline
+
+qa_pipeline = pipeline(
+    "question-answering",
+    model="henryk/bert-base-multilingual-cased-finetuned-polish-squad1",
+    tokenizer="henryk/bert-base-multilingual-cased-finetuned-polish-squad1"
+)
+
+qa_pipeline({
+    'context': "Warszawa jest największym miastem w Polsce pod względem liczby ludności i powierzchni",
+    'question': "Jakie jest największe miasto w Polsce?"})
+
+```
+
+# Output:
+
+```json
+{
+  "score": 0.9988,
+  "start": 0, 
+  "end": 8,
+  "answer": "Warszawa"
+}
+```
+
+## Contact
+
+Please do not hesitate to contact me via [LinkedIn](https://www.linkedin.com/in/henryk-borzymowski-0755a2167/) if you want to discuss or get access to the Polish version of SQuAD.
@@ -0,0 +1,96 @@
+---
+language: polish
+---
+
+# Multilingual + Polish SQuAD2.0
+
+This model is the multilingual model provided by the Google research team with a fine-tuned polish Q&A downstream task.
+
+## Details of the language model
+
+Language model ([**bert-base-multilingual-cased**](https://github.com/google-research/bert/blob/master/multilingual.md)):
+12-layer, 768-hidden, 12-heads, 110M parameters.
+Trained on cased text in the top 104 languages with the largest Wikipedias.
+
+## Details of the downstream task
+Using the `mtranslate` Python module, [**SQuAD2.0**](https://rajpurkar.github.io/SQuAD-explorer/) was machine-translated. In order to find the start tokens, the direct translations of the answers were searched in the corresponding paragraphs. Due to the different translations depending on the context (missing context in the pure answer), the answer could not always be found in the text, and thus a loss of question-answer examples occurred. This is a potential problem where errors can occur in the data set.
+
+| Dataset                | # Q&A |
+| ---------------------- | ----- |
+| SQuAD2.0 Train         | 130 K |
+| Polish SQuAD2.0 Train   | 83.1 K |
+| SQuAD2.0 Dev           |  12 K |
+| Polish SQuAD2.0 Dev     | 8.5  K |
+
+
+## Model benchmark
+
+| Model                | EM/F1 |HasAns (EM/F1) | NoAns |
+| ---------------------- | ----- | ----- | ----- |
+| [SlavicBERT](https://huggingface.co/DeepPavlov/bert-base-bg-cs-pl-ru-cased)   | 69.35/71.51  | 47.02/54.09 | 79.20 |
+| [polBERT](https://huggingface.co/dkleczek/bert-base-polish-uncased-v1)   | 67.33/69.80| 45.73/53.80  | 76.87 |
+| [multiBERT](https://huggingface.co/bert-base-multilingual-cased) | **70.76**/**72.92**  |45.00/52.04 | 82.13 |
+
+## Model training
+
+The model was trained on a **Tesla V100** GPU with the following command:
+
+```python
+export SQUAD_DIR=path/to/pl_squad
+
+python run_squad.py 
+  --model_type bert \
+  --model_name_or_path bert-base-multilingual-cased \
+  --do_train \
+  --do_eval \
+  --version_2_with_negative \
+  --train_file $SQUAD_DIR/pl_squadv2_train.json \
+  --predict_file $SQUAD_DIR/pl_squadv2_dev.json \
+  --num_train_epochs 2 \
+  --max_seq_length 384 \
+  --doc_stride 128 \
+  --save_steps=8000 \
+  --output_dir ../../output \
+  --overwrite_cache \
+  --overwrite_output_dir
+```
+
+**Results**:
+
+{'exact': 70.76671723655035, 'f1': 72.92156947155917, 'total': 8569, 'HasAns_exact': 45.00762195121951, 'HasAns_f1': 52.04456128116991, 'HasAns_total': 2624, 'NoAns_exact': 82.13624894869638, '
+NoAns_f1': 82.13624894869638, 'NoAns_total': 5945, 'best_exact': 71.72365503559342, 'best_exact_thresh': 0.0, 'best_f1': 73.62662512059369, 'best_f1_thresh': 0.0}
+
+
+## Model in action
+
+Fast usage with **pipelines**:
+
+```python
+from transformers import pipeline
+
+qa_pipeline = pipeline(
+    "question-answering",
+    model="henryk/bert-base-multilingual-cased-finetuned-polish-squad2",
+    tokenizer="henryk/bert-base-multilingual-cased-finetuned-polish-squad2"
+)
+
+qa_pipeline({
+    'context': "Warszawa jest największym miastem w Polsce pod względem liczby ludności i powierzchni",
+    'question': "Jakie jest największe miasto w Polsce?"})
+
+```
+
+# Output:
+
+```json
+{
+  "score": 0.9986,
+  "start": 0, 
+  "end": 8,
+  "answer": "Warszawa"
+}
+```
+
+## Contact
+
+Please do not hesitate to contact me via [LinkedIn](https://www.linkedin.com/in/henryk-borzymowski-0755a2167/) if you want to discuss or get access to the Polish version of SQuAD.
@@ -0,0 +1,86 @@
+---
+language: malay
+---
+
+# Bahasa Albert Model
+
+Pretrained Albert base language model for Malay and Indonesian. 
+
+## Pretraining Corpus
+
+`albert-base-bahasa-cased` model was pretrained on ~1.8 Billion words. We trained on both standard and social media language structures, and below is list of data we trained on,
+
+1. [dumping wikipedia](https://github.com/huseinzol05/Malaya-Dataset#wikipedia-1).
+2. [local instagram](https://github.com/huseinzol05/Malaya-Dataset#instagram).
+3. [local twitter](https://github.com/huseinzol05/Malaya-Dataset#twitter-1).
+4. [local news](https://github.com/huseinzol05/Malaya-Dataset#public-news).
+5. [local parliament text](https://github.com/huseinzol05/Malaya-Dataset#parliament).
+6. [local singlish/manglish text](https://github.com/huseinzol05/Malaya-Dataset#singlish-text).
+7. [IIUM Confession](https://github.com/huseinzol05/Malaya-Dataset#iium-confession).
+8. [Wattpad](https://github.com/huseinzol05/Malaya-Dataset#wattpad).
+9. [Academia PDF](https://github.com/huseinzol05/Malaya-Dataset#academia-pdf).
+
+Preprocessing steps can reproduce from here, [Malaya/pretrained-model/preprocess](https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/preprocess).
+
+## Pretraining details
+
+- This model was trained using Google Albert's github [repository](https://github.com/google-research/ALBERT) on v3-8 TPU.
+- All steps can reproduce from here, [Malaya/pretrained-model/albert](https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/albert).
+
+## Load Pretrained Model
+
+You can use this model by installing `torch` or `tensorflow` and Huggingface library `transformers`. And you can use it directly by initializing it like this:  
+
+```python
+from transformers import AlbertTokenizer, AlbertModel
+
+model = BertModel.from_pretrained('huseinzol05/albert-base-bahasa-cased')
+tokenizer = AlbertTokenizer.from_pretrained(
+    'huseinzol05/albert-base-bahasa-cased',
+    do_lower_case = False,
+)
+```
+
+## Example using AutoModelWithLMHead
+
+```python
+from transformers import AlbertTokenizer, AutoModelWithLMHead, pipeline
+
+model = AutoModelWithLMHead.from_pretrained('huseinzol05/albert-base-bahasa-cased')
+tokenizer = AlbertTokenizer.from_pretrained(
+    'huseinzol05/albert-base-bahasa-cased',
+    do_lower_case = False,
+)
+fill_mask = pipeline('fill-mask', model = model, tokenizer = tokenizer)
+print(fill_mask('makan ayam dengan [MASK]'))
+```
+
+Output is,
+
+```text
+[{'sequence': '[CLS] makan ayam dengan ayam[SEP]',
+  'score': 0.044952988624572754,
+  'token': 629},
+ {'sequence': '[CLS] makan ayam dengan sayur[SEP]',
+  'score': 0.03621877357363701,
+  'token': 1639},
+ {'sequence': '[CLS] makan ayam dengan ikan[SEP]',
+  'score': 0.034429922699928284,
+  'token': 758},
+ {'sequence': '[CLS] makan ayam dengan nasi[SEP]',
+  'score': 0.032447945326566696,
+  'token': 453},
+ {'sequence': '[CLS] makan ayam dengan rendang[SEP]',
+  'score': 0.028885239735245705,
+  'token': 2451}]
+```
+
+## Results
+
+For further details on the model performance, simply checkout accuracy page from Malaya, https://malaya.readthedocs.io/en/latest/Accuracy.html, we compared with traditional models.
+
+## Acknowledgement
+
+Thanks to [Im Big](https://www.facebook.com/imbigofficial/), [LigBlou](https://www.facebook.com/ligblou), [Mesolitica](https://mesolitica.com/) and [KeyReply](https://www.keyreply.com/) for sponsoring AWS, Google and GPU clouds to train Albert for Bahasa. 
+
+
@@ -0,0 +1,86 @@
+---
+language: malay
+---
+
+# Bahasa Albert Model
+
+Pretrained Albert tiny language model for Malay and Indonesian, 85% faster execution and 50% smaller than Albert base.
+
+## Pretraining Corpus
+
+`albert-tiny-bahasa-cased` model was pretrained on ~1.8 Billion words. We trained on both standard and social media language structures, and below is list of data we trained on,
+
+1. [dumping wikipedia](https://github.com/huseinzol05/Malaya-Dataset#wikipedia-1).
+2. [local instagram](https://github.com/huseinzol05/Malaya-Dataset#instagram).
+3. [local twitter](https://github.com/huseinzol05/Malaya-Dataset#twitter-1).
+4. [local news](https://github.com/huseinzol05/Malaya-Dataset#public-news).
+5. [local parliament text](https://github.com/huseinzol05/Malaya-Dataset#parliament).
+6. [local singlish/manglish text](https://github.com/huseinzol05/Malaya-Dataset#singlish-text).
+7. [IIUM Confession](https://github.com/huseinzol05/Malaya-Dataset#iium-confession).
+8. [Wattpad](https://github.com/huseinzol05/Malaya-Dataset#wattpad).
+9. [Academia PDF](https://github.com/huseinzol05/Malaya-Dataset#academia-pdf).
+
+Preprocessing steps can reproduce from here, [Malaya/pretrained-model/preprocess](https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/preprocess).
+
+## Pretraining details
+
+- This model was trained using Google Albert's github [repository](https://github.com/google-research/ALBERT) on v3-8 TPU.
+- All steps can reproduce from here, [Malaya/pretrained-model/albert](https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/albert).
+
+## Load Pretrained Model
+
+You can use this model by installing `torch` or `tensorflow` and Huggingface library `transformers`. And you can use it directly by initializing it like this:  
+
+```python
+from transformers import AlbertTokenizer, AlbertModel
+
+model = BertModel.from_pretrained('huseinzol05/albert-tiny-bahasa-cased')
+tokenizer = AlbertTokenizer.from_pretrained(
+    'huseinzol05/albert-tiny-bahasa-cased',
+    do_lower_case = False,
+)
+```
+
+## Example using AutoModelWithLMHead
+
+```python
+from transformers import AlbertTokenizer, AutoModelWithLMHead, pipeline
+
+model = AutoModelWithLMHead.from_pretrained('huseinzol05/albert-tiny-bahasa-cased')
+tokenizer = AlbertTokenizer.from_pretrained(
+    'huseinzol05/albert-tiny-bahasa-cased',
+    do_lower_case = False,
+)
+fill_mask = pipeline('fill-mask', model = model, tokenizer = tokenizer)
+print(fill_mask('makan ayam dengan [MASK]'))
+```
+
+Output is,
+
+```text
+[{'sequence': '[CLS] makan ayam dengan ayam[SEP]',
+  'score': 0.05121927708387375,
+  'token': 629},
+ {'sequence': '[CLS] makan ayam dengan sayur[SEP]',
+  'score': 0.04497420787811279,
+  'token': 1639},
+ {'sequence': '[CLS] makan ayam dengan nasi[SEP]',
+  'score': 0.039827536791563034,
+  'token': 453},
+ {'sequence': '[CLS] makan ayam dengan rendang[SEP]',
+  'score': 0.032997727394104004,
+  'token': 2451},
+ {'sequence': '[CLS] makan ayam dengan makan[SEP]',
+  'score': 0.031354598701000214,
+  'token': 129}]
+```
+
+## Results
+
+For further details on the model performance, simply checkout accuracy page from Malaya, https://malaya.readthedocs.io/en/latest/Accuracy.html, we compared with traditional models.
+
+## Acknowledgement
+
+Thanks to [Im Big](https://www.facebook.com/imbigofficial/), [LigBlou](https://www.facebook.com/ligblou), [Mesolitica](https://mesolitica.com/) and [KeyReply](https://www.keyreply.com/) for sponsoring AWS, Google and GPU clouds to train Albert for Bahasa. 
+
+
@@ -4,7 +4,7 @@ language: malay

 # Bahasa Tiny-BERT Model

-General Distilled Tiny BERT base language model for Malay and Indonesian. 
+General Distilled Tiny BERT language model for Malay and Indonesian. 

 ## Pretraining Corpus

@@ -36,7 +36,7 @@ from transformers import AlbertTokenizer, BertModel

 model = BertModel.from_pretrained('huseinzol05/tiny-bert-bahasa-cased')
 tokenizer = AlbertTokenizer.from_pretrained(
-    'huseinzol05/tiny-base-bahasa-cased',
+    'huseinzol05/tiny-bert-bahasa-cased',
    unk_token = '[UNK]',
    pad_token = '[PAD]',
    do_lower_case = False,
@@ -50,9 +50,9 @@ We use [google/sentencepiece](https://github.com/google/sentencepiece) to train
 ```python
 from transformers import AlbertTokenizer, AutoModelWithLMHead, pipeline

-model = AutoModelWithLMHead.from_pretrained('huseinzol05/tiny-base-bahasa-cased')
+model = AutoModelWithLMHead.from_pretrained('huseinzol05/tiny-bert-bahasa-cased')
 tokenizer = AlbertTokenizer.from_pretrained(
-    'huseinzol05/tiny-base-bahasa-cased',
+    'huseinzol05/tiny-bert-bahasa-cased',
    unk_token = '[UNK]',
    pad_token = '[PAD]',
    do_lower_case = False,
@@ -0,0 +1,29 @@
+---
+language:
+- basque
+---
+
+# BERTeus base cased
+
+This is the Basque language pretrained model presented in [Give your Text Representation Models some Love: the Case for Basque](https://arxiv.org/pdf/2004.00033.pdf). This model has been trained on a Basque corpus comprising Basque crawled news articles from online newspapers and the Basque Wikipedia. The training corpus contains 224.6 million tokens, of which 35 million come from the Wikipedia.
+
+BERTeus has been tested on four different downstream tasks for Basque: part-of-speech (POS) tagging, named entity recognition (NER), sentiment analysis and topic classification; improving the state of the art for all tasks. See summary of results below:
+
+
+| Downstream task | BERTeus | mBERT | Previous SOTA |
+| --------------- | ------- | ------| ------------- |
+| Topic Classification	  | **76.77**   | 68.42 | 63.00 	    |
+| Sentiment    	  | **78.10**   | 71.02 | 74.02 	    |
+| POS   	  | **97.76**   | 96.37 | 96.10 	    |
+| NER    	  | **87.06**   | 81.52 | 76.72 	    |
+
+
+If using this model, please cite the following paper:
+```
+@inproceedings{agerri2020give,
+  title={Give your Text Representation Models some Love: the Case for Basque},
+  author={Rodrigo Agerri and I{\~n}aki San Vicente and Jon Ander Campos and Ander Barrena and Xabier Saralegi and Aitor Soroa and Eneko Agirre},
+  booktitle={Proceedings of the 12th International Conference on Language Resources and Evaluation},
+  year={2020}
+}
+```
@@ -0,0 +1,61 @@
+### Model
+**[`albert-xlarge-v2`](https://huggingface.co/albert-xlarge-v2)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)**
+
+### Training Parameters
+Trained on 4 NVIDIA GeForce RTX 2080 Ti 11Gb
+```bash
+BASE_MODEL=albert-xlarge-v2
+python run_squad.py \
+  --version_2_with_negative \
+  --model_type albert \
+  --model_name_or_path $BASE_MODEL \
+  --output_dir $OUTPUT_MODEL \
+  --do_eval \
+  --do_lower_case \
+  --train_file $SQUAD_DIR/train-v2.0.json \
+  --predict_file $SQUAD_DIR/dev-v2.0.json \
+  --per_gpu_train_batch_size 3 \
+  --per_gpu_eval_batch_size 64 \
+  --learning_rate 3e-5 \
+  --num_train_epochs 3.0 \
+  --max_seq_length 384 \
+  --doc_stride 128 \
+  --save_steps 2000 \
+  --threads 24 \
+  --warmup_steps 814 \
+  --gradient_accumulation_steps 4 \
+  --fp16 \
+  --do_train
+```
+  
+### Evaluation
+
+Evaluation on the dev set. I did not sweep for best threshold.
+
+|                   | val               |
+|-------------------|-------------------|
+| exact             | 84.41842836688285 |
+| f1                | 87.4628460501696  |
+| total             | 11873.0           |
+| HasAns_exact      | 80.68488529014844 |
+| HasAns_f1         | 86.78245127423482 |
+| HasAns_total      | 5928.0            |
+| NoAns_exact       | 88.1412952060555  |
+| NoAns_f1          | 88.1412952060555  |
+| NoAns_total       | 5945.0            |
+| best_exact        | 84.41842836688285 |
+| best_exact_thresh | 0.0               |
+| best_f1           | 87.46284605016956 |
+| best_f1_thresh    | 0.0               |
+
+
+### Usage
+
+See [huggingface documentation](https://huggingface.co/transformers/model_doc/albert.html#albertforquestionanswering). Training on `SQuAD V2` allows the model to score if a paragraph contains an answer:
+```python
+start_scores, end_scores = model(input_ids) 
+span_scores = start_scores.softmax(dim=1).log()[:,:,None] + end_scores.softmax(dim=1).log()[:,None,:]
+ignore_score = span_scores[:,0,0] #no answer scores
+    
+```
+
@@ -0,0 +1,64 @@
+### Model
+**[`monologg/biobert_v1.1_pubmed`](https://huggingface.co/monologg/biobert_v1.1_pubmed)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)**
+
+This model is cased.
+
+### Training Parameters
+Trained on 4 NVIDIA GeForce RTX 2080 Ti 11Gb
+```bash
+BASE_MODEL=monologg/biobert_v1.1_pubmed
+python run_squad.py \
+  --version_2_with_negative \
+  --model_type albert \
+  --model_name_or_path $BASE_MODEL \
+  --output_dir $OUTPUT_MODEL \
+  --do_eval \
+  --do_lower_case \
+  --train_file $SQUAD_DIR/train-v2.0.json \
+  --predict_file $SQUAD_DIR/dev-v2.0.json \
+  --per_gpu_train_batch_size 18 \
+  --per_gpu_eval_batch_size 64 \
+  --learning_rate 3e-5 \
+  --num_train_epochs 3.0 \
+  --max_seq_length 384 \
+  --doc_stride 128 \
+  --save_steps 2000 \
+  --threads 24 \
+  --warmup_steps 550 \
+  --gradient_accumulation_steps 1 \
+  --fp16 \
+  --logging_steps 50 \
+  --do_train
+```
+  
+### Evaluation
+
+Evaluation on the dev set. I did not sweep for best threshold.
+
+|                   | val               |
+|-------------------|-------------------|
+| exact             | 75.97068980038743 |
+| f1                | 79.37043950121722 |
+| total             | 11873.0           |
+| HasAns_exact      | 74.13967611336032 |
+| HasAns_f1         | 80.94892513460755 |
+| HasAns_total      | 5928.0            |
+| NoAns_exact       | 77.79646761984861 |
+| NoAns_f1          | 77.79646761984861 |
+| NoAns_total       | 5945.0            |
+| best_exact        | 75.97068980038743 |
+| best_exact_thresh | 0.0               |
+| best_f1           | 79.37043950121729 |
+| best_f1_thresh    | 0.0               |
+
+
+### Usage
+
+See [huggingface documentation](https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering). Training on `SQuAD V2` allows the model to score if a paragraph contains an answer:
+```python
+start_scores, end_scores = model(input_ids) 
+span_scores = start_scores.softmax(dim=1).log()[:,:,None] + end_scores.softmax(dim=1).log()[:,None,:]
+ignore_score = span_scores[:,0,0] #no answer scores
+    
+```
+
@@ -0,0 +1,61 @@
+### Model
+**[`allenai/scibert_scivocab_uncased`](https://huggingface.co/allenai/scibert_scivocab_uncased)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)**
+
+### Training Parameters
+Trained on 4 NVIDIA GeForce RTX 2080 Ti 11Gb
+```bash
+BASE_MODEL=allenai/scibert_scivocab_uncased
+python run_squad.py \
+  --version_2_with_negative \
+  --model_type albert \
+  --model_name_or_path $BASE_MODEL \
+  --output_dir $OUTPUT_MODEL \
+  --do_eval \
+  --do_lower_case \
+  --train_file $SQUAD_DIR/train-v2.0.json \
+  --predict_file $SQUAD_DIR/dev-v2.0.json \
+  --per_gpu_train_batch_size 18 \
+  --per_gpu_eval_batch_size 64 \
+  --learning_rate 3e-5 \
+  --num_train_epochs 3.0 \
+  --max_seq_length 384 \
+  --doc_stride 128 \
+  --save_steps 2000 \
+  --threads 24 \
+  --warmup_steps 550 \
+  --gradient_accumulation_steps 1 \
+  --fp16 \
+  --logging_steps 50 \
+  --do_train
+```
+  
+### Evaluation
+
+Evaluation on the dev set. I did not sweep for best threshold.
+
+|                   | val               |
+|-------------------|-------------------|
+| exact             | 75.07790785816559 |
+| f1                | 78.47735207283013 |
+| total             | 11873.0           |
+| HasAns_exact      | 70.76585695006747 |
+| HasAns_f1         | 77.57449412292718 |
+| HasAns_total      | 5928.0            |
+| NoAns_exact       | 79.37762825904122 |
+| NoAns_f1          | 79.37762825904122 |
+| NoAns_total       | 5945.0            |
+| best_exact        | 75.08633032931863 |
+| best_exact_thresh | 0.0               |
+| best_f1           | 78.48577454398324 |
+| best_f1_thresh    | 0.0               |
+
+### Usage
+
+See [huggingface documentation](https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering). Training on `SQuAD V2` allows the model to score if a paragraph contains an answer:
+```python
+start_scores, end_scores = model(input_ids) 
+span_scores = start_scores.softmax(dim=1).log()[:,:,None] + end_scores.softmax(dim=1).log()[:,None,:]
+ignore_score = span_scores[:,0,0] #no answer scores
+    
+```
+
@@ -0,0 +1,27 @@
+# GPT2-IMDB-neg (LM + RL) 🎞😡✍
+
+All credits to [@lvwerra](https://twitter.com/lvwerra)
+
+## What is it?
+A small GPT2 (`lvwerra/gpt2-imdb`) language model fine-tuned to produce **negative** movie reviews based the [IMDB dataset](https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews). The model is trained with rewards from a BERT sentiment classifier (`lvwerra/gpt2-imdb`) via **PPO**.
+
+## Why?
+I wanted to reproduce the experiment [lvwerra/gpt2-imdb-pos](https://huggingface.co/lvwerra/gpt2-imdb-pos) but for generating **negative** movie reviews.
+
+## Training setting
+The model was trained for `100` optimisation steps with a batch size of `256` which corresponds to `25600` training samples. The full experiment setup (for positive samples) in [trl repo](https://lvwerra.github.io/trl/04-gpt2-sentiment-ppo-training/).
+
+## Examples
+A few examples of the model response to a query before and after optimisation:
+
+| query | response (before) | response (after) | rewards (before) | rewards (after) |
+|-------|-------------------|------------------|------------------|-----------------|
+|This movie is a fine |	attempt as far as live action is concerned, n...|example of how bad Hollywood in theatrics pla...|	2.118391 |	-3.31625|
+|I have watched 3 episodes |with this guy and he is such a talented actor...|	but the show is just plain awful and there ne...|	2.681171|	-4.512792|
+|We know that firefighters and|	police officers are forced to become populari...|	other chains have going to get this disaster ...|	1.367811|	-3.34017|
+
+
+
+> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
+
+> Made with <span style="color: #e25555;">&hearts;</span> in Spain
@@ -0,0 +1,80 @@
+---
+language: english
+thumbnail:
+---
+
+# SpanBERT base fine-tuned on SQuAD v1
+
+[SpanBERT](https://github.com/facebookresearch/SpanBERT) created by [Facebook Research](https://github.com/facebookresearch) and fine-tuned on [SQuAD 1.1](https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/) for **Q&A** downstream task ([by them](https://github.com/facebookresearch/SpanBERT#finetuned-models-squad-1120-relation-extraction-coreference-resolution)).
+
+## Details of SpanBERT
+
+[SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://arxiv.org/abs/1907.10529)
+
+## Details of the downstream task (Q&A) - Dataset 📚 🧐 ❓
+
+[SQuAD1.1](https://rajpurkar.github.io/SQuAD-explorer/)
+
+## Model fine-tuning 🏋️‍
+
+You can get the fine-tuning script [here](https://github.com/facebookresearch/SpanBERT)
+
+```bash
+python code/run_squad.py \
+  --do_train \
+  --do_eval \
+  --model spanbert-base-cased \
+  --train_file train-v1.1.json \
+  --dev_file dev-v1.1.json \
+  --train_batch_size 32 \
+  --eval_batch_size 32  \
+  --learning_rate 2e-5 \
+  --num_train_epochs 4 \
+  --max_seq_length 512 \
+  --doc_stride 128 \
+  --eval_metric f1 \
+  --output_dir squad_output \
+  --fp16
+```
+
+## Results Comparison 📝
+
+|                   | SQuAD 1.1     | SQuAD 2.0  | Coref   | TACRED |
+| ----------------------  | ------------- | ---------  | ------- | ------ |
+|                         | F1            | F1         | avg. F1 |  F1    |
+| BERT (base)             | 88.5         | 76.5     | 73.1    |  67.7  |
+| SpanBERT (base)         | **92.4** (this one)         | [83.6](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv2)      | 77.4    |  [68.2](https://huggingface.co/mrm8488/spanbert-base-finetuned-tacred)  |
+| BERT (large)            | 91.3          | 83.3       | 77.1    |  66.4  |
+| SpanBERT (large)        | [94.6](https://huggingface.co/mrm8488/spanbert-large-finetuned-squadv1)         | [88.7](https://huggingface.co/mrm8488/spanbert-large-finetuned-squadv2)     | 79.6    |  [70.8](https://huggingface.co/mrm8488/spanbert-large-finetuned-tacred)  |
+
+
+Note: The numbers marked as * are evaluated on the development sets becaus those models were not submitted to the official SQuAD leaderboard. All the other numbers are test numbers.
+
+## Model in action
+
+Fast usage with **pipelines**:
+
+```python
+from transformers import pipeline
+
+qa_pipeline = pipeline(
+    "question-answering",
+    model="mrm8488/spanbert-base-finetuned-squadv1",
+    tokenizer="SpanBERT/spanbert-base-cased"
+)
+
+qa_pipeline({
+    'context': "Manuel Romero has been working very hard in the repository hugginface/transformers lately",
+    'question': "How has been working Manuel Romero lately?"
+
+})
+
+# Output: {'answer': 'very hard in the repository hugginface/transformers',
+ 'end': 82,
+ 'score': 0.327230326857725,
+ 'start': 31}
+```
+
+> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
+
+> Made with <span style="color: #e25555;">&hearts;</span> in Spain
@@ -0,0 +1,82 @@
+---
+language: english
+thumbnail:
+---
+
+# SpanBERT base fine-tuned on SQuAD v2
+
+[SpanBERT](https://github.com/facebookresearch/SpanBERT) created by [Facebook Research](https://github.com/facebookresearch) and fine-tuned on [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) for **Q&A** downstream task ([by them](https://github.com/facebookresearch/SpanBERT#finetuned-models-squad-1120-relation-extraction-coreference-resolution)).
+
+## Details of SpanBERT
+
+[SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://arxiv.org/abs/1907.10529)
+
+## Details of the downstream task (Q&A) - Dataset 📚 🧐 ❓
+
+[SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.
+
+| Dataset  | Split | # samples |
+| -------- | ----- | --------- |
+| SQuAD2.0 | train | 130k      |
+| SQuAD2.0 | eval  | 12.3k     |
+
+## Model fine-tuning 🏋️‍
+
+You can get the fine-tuning script [here](https://github.com/facebookresearch/SpanBERT)
+
+```bash
+python code/run_squad.py \
+  --do_train \
+  --do_eval \
+  --model spanbert-base-cased \
+  --train_file train-v2.0.json \
+  --dev_file dev-v2.0.json \
+  --train_batch_size 32 \
+  --eval_batch_size 32  \
+  --learning_rate 2e-5 \
+  --num_train_epochs 4 \
+  --max_seq_length 512 \
+  --doc_stride 128 \
+  --eval_metric best_f1 \
+  --output_dir squad2_output \
+  --version_2_with_negative \
+  --fp16
+```
+
+## Results Comparison 📝
+
+|                   | SQuAD 1.1     | SQuAD 2.0  | Coref   | TACRED |
+| ----------------------  | ------------- | ---------  | ------- | ------ |
+|                         | F1            | F1         | avg. F1 |  F1    |
+| BERT (base)             | 88.5         | 76.5      | 73.1    |  67.7  |
+| SpanBERT (base)         | [92.4](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv1)         | **83.6** (this one)      | 77.4    |  [68.2](https://huggingface.co/mrm8488/spanbert-base-finetuned-tacred)  |
+| BERT (large)            | 91.3          | 83.3       | 77.1    |  66.4  |
+| SpanBERT (large)        | [94.6](https://huggingface.co/mrm8488/spanbert-large-finetuned-squadv1)          | [88.7](https://huggingface.co/mrm8488/spanbert-large-finetuned-squadv2)     | 79.6    |  [70.8](https://huggingface.co/mrm8488/spanbert-large-finetuned-tacred)  |
+
+
+Note: The numbers marked as * are evaluated on the development sets becaus those models were not submitted to the official SQuAD leaderboard. All the other numbers are test numbers.
+
+## Model in action
+
+Fast usage with **pipelines**:
+
+```python
+from transformers import pipeline
+
+qa_pipeline = pipeline(
+    "question-answering",
+    model="mrm8488/spanbert-base-finetuned-squadv2",
+    tokenizer="SpanBERT/spanbert-base-cased"
+)
+
+qa_pipeline({
+    'context': "Manuel Romero has been working very hard in the repository hugginface/transformers lately",
+    'question': "How has been working Manuel Romero lately?"
+
+})
+# Output: {'answer': 'very hard', 'end': 40, 'score': 0.9052708846768347, 'start': 31}
+```
+
+> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
+
+> Made with <span style="color: #e25555;">&hearts;</span> in Spain
@@ -0,0 +1,53 @@
+---
+language: english
+thumbnail:
+---
+
+# SpanBERT base fine-tuned on TACRED
+
+[SpanBERT](https://github.com/facebookresearch/SpanBERT) created by [Facebook Research](https://github.com/facebookresearch) and fine-tuned on [TACRED](https://nlp.stanford.edu/projects/tacred/) dataset by [them](https://github.com/facebookresearch/SpanBERT#finetuned-models-squad-1120-relation-extraction-coreference-resolution)
+
+## Details of SpanBERT
+
+[SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://arxiv.org/abs/1907.10529)
+
+## Dataset 📚
+
+[TACRED](https://nlp.stanford.edu/projects/tacred/) A large-scale relation extraction dataset with 106k+ examples over 42 TAC KBP relation types.
+
+## Model fine-tuning 🏋️‍
+
+You can get the fine-tuning script [here](https://github.com/facebookresearch/SpanBERT)
+
+```bash
+python code/run_tacred.py \
+  --do_train \
+  --do_eval \
+  --data_dir <TACRED_DATA_DIR> \
+  --model spanbert-base-cased \
+  --train_batch_size 32 \
+  --eval_batch_size 32 \
+  --learning_rate 2e-5 \
+  --num_train_epochs 10 \
+  --max_seq_length 128 \
+  --output_dir tacred_dir \
+  --fp16
+```
+
+## Results Comparison 📝
+
+|                   | SQuAD 1.1     | SQuAD 2.0  | Coref   | TACRED |
+| ----------------------  | ------------- | ---------  | ------- | ------ |
+|                         | F1            | F1         | avg. F1 |  F1    |
+| BERT (base)             | 88.5*         | 76.5*      | 73.1    |  67.7  |
+| SpanBERT (base)         | [92.4*](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv1)         | [83.6*](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv2)      | 77.4    |  **68.2** (this one)  |
+| BERT (large)            | 91.3          | 83.3       | 77.1    |  66.4  |
+| SpanBERT (large)        | [94.6](https://huggingface.co/mrm8488/spanbert-large-finetuned-squadv1)        | [88.7](https://huggingface.co/mrm8488/spanbert-large-finetuned-squadv2)     | 79.6    |  [70.8](https://huggingface.co/mrm8488/spanbert-base-finetuned-tacred)   |
+
+
+Note: The numbers marked as * are evaluated on the development sets becaus those models were not submitted to the official SQuAD leaderboard. All the other numbers are test numbers.
+
+
+> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
+
+> Made with <span style="color: #e25555;">&hearts;</span> in Spain
@@ -0,0 +1,80 @@
+---
+language: english
+thumbnail:
+---
+
+# SpanBERT large fine-tuned on SQuAD v1
+
+[SpanBERT](https://github.com/facebookresearch/SpanBERT) created by [Facebook Research](https://github.com/facebookresearch) and fine-tuned on [SQuAD 1.1](https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/) for **Q&A** downstream task ([by them](https://github.com/facebookresearch/SpanBERT#finetuned-models-squad-1120-relation-extraction-coreference-resolution)).
+
+## Details of SpanBERT
+
+[SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://arxiv.org/abs/1907.10529)
+
+## Details of the downstream task (Q&A) - Dataset 📚 🧐 ❓
+
+[SQuAD1.1](https://rajpurkar.github.io/SQuAD-explorer/)
+
+## Model fine-tuning 🏋️‍
+
+You can get the fine-tuning script [here](https://github.com/facebookresearch/SpanBERT)
+
+```bash
+python code/run_squad.py \
+  --do_train \
+  --do_eval \
+  --model spanbert-large-cased \
+  --train_file train-v1.1.json \
+  --dev_file dev-v1.1.json \
+  --train_batch_size 32 \
+  --eval_batch_size 32  \
+  --learning_rate 2e-5 \
+  --num_train_epochs 4 \
+  --max_seq_length 512 \
+  --doc_stride 128 \
+  --eval_metric f1 \
+  --output_dir squad_output \
+  --fp16
+```
+
+## Results Comparison 📝
+
+|                   | SQuAD 1.1     | SQuAD 2.0  | Coref   | TACRED |
+| ----------------------  | ------------- | ---------  | ------- | ------ |
+|                         | F1            | F1         | avg. F1 |  F1    |
+| BERT (base)             | 88.5*         | 76.5*      | 73.1    |  67.7  |
+| SpanBERT (base)         | [92.4*](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv1)         | [83.6*](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv2)      | 77.4    |  [68.2](https://huggingface.co/mrm8488/spanbert-base-finetuned-tacred)  |
+| BERT (large)            | 91.3          | 83.3       | 77.1    |  66.4  |
+| SpanBERT (large)        | **94.6** (this)         | [88.7](https://huggingface.co/mrm8488/spanbert-large-finetuned-squadv2)     | 79.6    |  [70.8](https://huggingface.co/mrm8488/spanbert-large-finetuned-tacred)  |
+
+
+Note: The numbers marked as * are evaluated on the development sets becaus those models were not submitted to the official SQuAD leaderboard. All the other numbers are test numbers.
+
+## Model in action
+
+Fast usage with **pipelines**:
+
+```python
+from transformers import pipeline
+
+qa_pipeline = pipeline(
+    "question-answering",
+    model="mrm8488/spanbert-large-finetuned-squadv1",
+    tokenizer="SpanBERT/spanbert-large-cased"
+)
+
+qa_pipeline({
+    'context': "Manuel Romero has been working very hard in the repository hugginface/transformers lately",
+    'question': "How has been working Manuel Romero lately?"
+
+})
+
+# Output: {'answer': 'very hard in the repository hugginface/transformers',
+ 'end': 82,
+ 'score': 0.327230326857725,
+ 'start': 31}
+```
+
+> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
+
+> Made with <span style="color: #e25555;">&hearts;</span> in Spain
@@ -0,0 +1,82 @@
+---
+language: english
+thumbnail:
+---
+
+# SpanBERT large fine-tuned on SQuAD v2
+
+[SpanBERT](https://github.com/facebookresearch/SpanBERT) created by [Facebook Research](https://github.com/facebookresearch) and fine-tuned on [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) for **Q&A** downstream task ([by them](https://github.com/facebookresearch/SpanBERT#finetuned-models-squad-1120-relation-extraction-coreference-resolution)).
+
+## Details of SpanBERT
+
+[SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://arxiv.org/abs/1907.10529)
+
+## Details of the downstream task (Q&A) - Dataset 📚 🧐 ❓
+
+[SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.
+
+| Dataset  | Split | # samples |
+| -------- | ----- | --------- |
+| SQuAD2.0 | train | 130k      |
+| SQuAD2.0 | eval  | 12.3k     |
+
+## Model fine-tuning 🏋️‍
+
+You can get the fine-tuning script [here](https://github.com/facebookresearch/SpanBERT)
+
+```bash
+python code/run_squad.py \
+  --do_train \
+  --do_eval \
+  --model spanbert-large-cased \
+  --train_file train-v2.0.json \
+  --dev_file dev-v2.0.json \
+  --train_batch_size 32 \
+  --eval_batch_size 32  \
+  --learning_rate 2e-5 \
+  --num_train_epochs 4 \
+  --max_seq_length 512 \
+  --doc_stride 128 \
+  --eval_metric best_f1 \
+  --output_dir squad2_output \
+  --version_2_with_negative \
+  --fp16
+```
+
+## Results Comparison 📝
+
+|                   | SQuAD 1.1     | SQuAD 2.0  | Coref   | TACRED |
+| ----------------------  | ------------- | ---------  | ------- | ------ |
+|                         | F1            | F1         | avg. F1 |  F1    |
+| BERT (base)             | 88.5*         | 76.5*      | 73.1    |  67.7  |
+| SpanBERT (base)         | [92.4*](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv1)         | [83.6*](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv2)      | 77.4    |  [68.2](https://huggingface.co/mrm8488/spanbert-base-finetuned-tacred)  |
+| BERT (large)            | 91.3          | 83.3       | 77.1    |  66.4  |
+| SpanBERT (large)        | [94.6](https://huggingface.co/mrm8488/spanbert-large-finetuned-squadv1)          | **88.7** (this)     | 79.6    |  [70.8](https://huggingface.co/mrm8488/spanbert-large-finetuned-tacred)  |
+
+
+Note: The numbers marked as * are evaluated on the development sets becaus those models were not submitted to the official SQuAD leaderboard. All the other numbers are test numbers.
+
+## Model in action
+
+Fast usage with **pipelines**:
+
+```python
+from transformers import pipeline
+
+qa_pipeline = pipeline(
+    "question-answering",
+    model="mrm8488/spanbert-large-finetuned-squadv2",
+    tokenizer="SpanBERT/spanbert-large-cased"
+)
+
+qa_pipeline({
+    'context': "Manuel Romero has been working very hard in the repository hugginface/transformers lately",
+    'question': "How has been working Manuel Romero lately?"
+
+})
+# Output: {'answer': 'very hard', 'end': 40, 'score': 0.9052708846768347, 'start': 31}
+```
+
+> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
+
+> Made with <span style="color: #e25555;">&hearts;</span> in Spain
@@ -0,0 +1,53 @@
+---
+language: english
+thumbnail:
+---
+
+# SpanBERT large fine-tuned on TACRED
+
+[SpanBERT](https://github.com/facebookresearch/SpanBERT) created by [Facebook Research](https://github.com/facebookresearch) and fine-tuned on [TACRED](https://nlp.stanford.edu/projects/tacred/) dataset by [them](https://github.com/facebookresearch/SpanBERT#finetuned-models-squad-1120-relation-extraction-coreference-resolution)
+
+## Details of SpanBERT
+
+[SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://arxiv.org/abs/1907.10529)
+
+## Dataset 📚
+
+[TACRED](https://nlp.stanford.edu/projects/tacred/) A large-scale relation extraction dataset with 106k+ examples over 42 TAC KBP relation types.
+
+## Model fine-tuning 🏋️‍
+
+You can get the fine-tuning script [here](https://github.com/facebookresearch/SpanBERT)
+
+```bash
+python code/run_tacred.py \
+  --do_train \
+  --do_eval \
+  --data_dir <TACRED_DATA_DIR> \
+  --model spanbert-large-cased \
+  --train_batch_size 32 \
+  --eval_batch_size 32 \
+  --learning_rate 2e-5 \
+  --num_train_epochs 10 \
+  --max_seq_length 128 \
+  --output_dir tacred_dir \
+  --fp16
+```
+
+## Results Comparison 📝
+
+|                   | SQuAD 1.1     | SQuAD 2.0  | Coref   | TACRED |
+| ----------------------  | ------------- | ---------  | ------- | ------ |
+|                         | F1            | F1         | avg. F1 |  F1    |
+| BERT (base)             | 88.5*         | 76.5*      | 73.1    |  67.7  |
+| SpanBERT (base)         | [92.4*](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv1)         | [83.6*](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv2)      | 77.4    |  [68.2](https://huggingface.co/mrm8488/spanbert-base-finetuned-tacred)  |
+| BERT (large)            | 91.3          | 83.3       | 77.1    |  66.4  |
+| SpanBERT (large)        | [94.6](https://huggingface.co/mrm8488/spanbert-large-finetuned-squadv1)        | [88.7](https://huggingface.co/mrm8488/spanbert-large-finetuned-squadv2)     | 79.6    |  **70.8** (this one)  |
+
+
+Note: The numbers marked as * are evaluated on the development sets becaus those models were not submitted to the official SQuAD leaderboard. All the other numbers are test numbers.
+
+
+> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
+
+> Made with <span style="color: #e25555;">&hearts;</span> in Spain
@@ -0,0 +1,18 @@
+## Reformer Model trained on "Crime and Punishment" 
+Crime and Punishment text was taken from `gs://trax-ml/reformer/crime-and-punishment-2554.txt`.
+Model was trained in flax using colab notebook proposed by authors: https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/text_generation.ipynb
+Weights were converted to Hugging Face PyTorch `ReformerModelWithLMHead`.
+
+Model is used as a proof of concept that the forward pass works for a `ReformerModelWithLMHead`.
+Given that the model was trained only for 30mins on a ~0.5M tokens dataset and has only 320 tokens, 
+the generation results are reasonable:
+
+```python
+model = ReformerModelWithLMHead.from_pretrained("patrickvonplaten/reformer-crime-and-punish")
+tok = ReformerTokenizer.from_pretrained("patrickvonplaten/reformer-crime-and-punish")
+tok.decode(model.generate(tok.encode("A few months later", return_tensors="pt"), do_sample=True,temperature=0.7, max_length=100)[0])
+
+# gives:'A few months later on was more than anything in the flat. 
+# “I have already.” “That’s not my notion that he had forgotten him. 
+# What does that matter? And why do you mean? It’s only another fellow,” he said as he went out, as though he want'
+```
@@ -0,0 +1,57 @@
+---
+language: german
+---
+
+# Model description
+## Dataset
+Trained on fictional and non-fictional German texts written between 1840 and 1920:
+* Narrative texts from Digitale Bibliothek (https://textgrid.de/digitale-bibliothek)
+* Fairy tales and sagas from Grimm Korpus (https://www1.ids-mannheim.de/kl/projekte/korpora/archiv/gri.html)
+* Newspaper and magazine article from Mannheimer Korpus Historischer Zeitungen und Zeitschriften (https://repos.ids-mannheim.de/mkhz-beschreibung.html)
+* Magazine article from the journal „Die Grenzboten“ (http://www.deutschestextarchiv.de/doku/textquellen#grenzboten)
+* Fictional and non-fictional texts from Projekt Gutenberg (https://www.projekt-gutenberg.org)
+
+## Hardware used
+1 Tesla P4 GPU
+
+## Hyperparameters
+
+| Parameter                     | Value    |
+|-------------------------------|----------|
+| Epochs                        | 3        |
+| Gradient_accumulation_steps   | 1        |
+| Train_batch_size              | 32       |
+| Learning_rate                 | 0.00003  |
+| Max_seq_len                   | 128      |
+
+## Evaluation results: Automatic tagging of four forms of speech/thought/writing representation in historical fictional and non-fictional German texts
+
+The language model was used in the task to tag direct, indirect, reported and free indirect speech/thought/writing representation in fictional and non-fictional German texts. The tagger is available and described in detail at https://github.com/redewiedergabe/tagger.
+
+The tagging model was trained using the SequenceTagger Class of the Flair framework ([Akbik et al., 2019](https://www.aclweb.org/anthology/N19-4010)) which implements a BiLSTM-CRF architecture on top of a language embedding (as proposed by [Huang et al. (2015)](https://arxiv.org/abs/1508.01991)). 
+
+
+Hyperparameters
+
+| Parameter                     | Value      |
+|-------------------------------|------------|
+| Hidden_size                   | 256        |
+| Learning_rate                 | 0.1        |
+| Mini_batch_size               | 8          |
+| Max_epochs                    | 150        |
+
+Results are reported below in comparison to a custom trained flair embedding, which was stacked onto a custom trained fastText-model. Both models were trained on the same dataset.
+
+|                | BERT       ||| FastText+Flair  |||Test data|
+|----------------|----------|-----------|----------|------|-----------|--------|--------|
+|                | F1       | Precision | Recall   | F1   | Precision | Recall ||
+| Direct         | 0.80     | 0.86      | 0.74     | 0.84 | 0.90      | 0.79   |historical German, fictional & non-fictional|
+| Indirect       | **0.76** | **0.79**  | **0.73** | 0.73 | 0.78      | 0.68   |historical German, fictional & non-fictional|
+| Reported       | **0.58** | **0.69**  | **0.51** | 0.56 | 0.68      | 0.48   |historical German, fictional & non-fictional|
+| Free indirect  | **0.57** | **0.80**  | **0.44** | 0.47 | 0.78      | 0.34   |modern German, fictional|
+
+## Intended use:
+Historical German Texts (1840 to 1920)
+
+(Showed good performance with modern German fictional texts as well)
+
@@ -0,0 +1,8 @@
+---
+tags:
+- exbert
+---
+
+<a href="https://huggingface.co/exbert/?model=roberta-base">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
@@ -0,0 +1,19 @@
+---
+tags: 
+- chemistry
+---
+
+# ChemBERTa: Training a BERT-like transformer model for masked language modelling of chemical SMILES strings.
+
+Deep learning for chemistry and materials science remains a novel field with lots of potiential. However, the popularity of transfer learning based methods in areas such as NLP and computer vision have not yet been effectively developed in computational chemistry + machine learning. Using HuggingFace's suite of models and the ByteLevel tokenizer, we are able to train on a large corpus of 100k SMILES strings from a commonly known benchmark dataset, ZINC.
+
+Training RoBERTa over 5 epochs, the model achieves a decent loss of 0.398, but may likely continue to decline if trained for a larger number of epochs. The model can predict tokens within a SMILES sequence/molecule, allowing for variants of a molecule within discoverable chemical space to be predicted.
+
+By applying the representations of functional groups and atoms learned by the model, we can try to tackle problems of toxicity, solubility, drug-likeness, and synthesis accessibility on smaller datasets using the learned representations as features for graph convolution and attention models on the graph structure of molecules, as well as fine-tuning of BERT. Finally, we propose the use of attention visualization as a helpful tool for chemistry practitioners and students to quickly identify important substructures in various chemical properties.
+
+Additionally, visualization of the attention mechanism have been seen through previous research as incredibly valuable towards chemical reaction classification. The applications of open-sourcing large-scale transformer models such as RoBERTa with HuggingFace may allow for the acceleration of these individual research directions.
+
+A link to a repository which includes the training, uploading and evaluation notebook (with sample predictions on compounds such as Remdesivir) can be found [here](https://github.com/seyonechithrananda/bert-loves-chemistry). All of the notebooks can be copied into a new Colab runtime for easy execution.
+
+Thanks for checking this out!
+- Seyone
@@ -0,0 +1,6 @@
+---
+tags:
+- summarization
+- translation
+---
+
--- a/Show More
+++ b/Show More