From d6bbcbc4cf79f0d6da6d4753f4d128ff7e3e42a5 Mon Sep 17 00:00:00 2001 From: Lysandre Date: Wed, 21 Aug 2019 11:22:05 -0400 Subject: [PATCH] Added finetuning example to documentation --- docs/source/examples.rst | 49 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/docs/source/examples.rst b/docs/source/examples.rst index 51c8d850..40e22725 100644 --- a/docs/source/examples.rst +++ b/docs/source/examples.rst @@ -12,8 +12,8 @@ Examples - How to use gradient-accumulation, multi-gpu training, distributed training, optimize on CPU and 16-bits training to train Bert models * - `Fine-tuning with BERT: running the examples <#fine-tuning-bert-examples>`_ - Running the examples in `examples `_\ : ``extract_classif.py``\ , ``run_bert_classifier.py``\ , ``run_bert_squad.py`` and ``run_lm_finetuning.py`` - * - `Fine-tuning with OpenAI GPT, Transformer-XL and GPT-2 <#fine-tuning>`_ - - Running the examples in `examples `_\ : ``run_openai_gpt.py``\ , ``run_transfo_xl.py`` and ``run_gpt2.py`` + * - `Fine-tuning with OpenAI GPT, Transformer-XL, GPT-2 as well as BERT and RoBERTa <#fine-tuning>`_ + - Running the examples in `examples `_\ : ``run_openai_gpt.py``\ , ``run_transfo_xl.py``, ``run_gpt2.py`` and ``run_lm_finetuning.py`` * - `Fine-tuning BERT-large on GPUs <#fine-tuning-bert-large>`_ - How to fine tune ``BERT large`` @@ -393,12 +393,13 @@ Thank to the work of @Rocketknight1 and @tholor there are now **several scripts* OpenAI GPT, Transformer-XL and GPT-2: running the examples ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -We provide three examples of scripts for OpenAI GPT, Transformer-XL and OpenAI GPT-2 based on (and extended from) the respective original implementations: +We provide three examples of scripts for OpenAI GPT, Transformer-XL, OpenAI GPT-2, BERT and RoBERTa based on (and extended from) the respective original implementations: * fine-tuning OpenAI GPT on the ROCStories dataset * evaluating Transformer-XL on Wikitext 103 * unconditional and conditional generation from a pre-trained OpenAI GPT-2 model +* fine-tuning GPT/GPT-2 on a causal language modeling task and BERT/RoBERTa on a masked language modeling task Fine-tuning OpenAI GPT on the RocStories dataset ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -452,7 +453,47 @@ Unconditional generation: python run_gpt2.py --unconditional -The same option as in the original scripts are provided, please refere to the code of the example and the original repository of OpenAI. +The same option as in the original scripts are provided, please refer to the code of the example and the original repository of OpenAI. + + +Causal LM fine-tuning on GPT/GPT-2, Masked LM fine-tuning on BERT/RoBERTa +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Before running the following examples you should download the `WikiText-2 dataset `__ and unpack it to some directory `$WIKITEXT_2_DATASET` +The following results were obtained using the `raw` WikiText-2 (no tokens were replaced before the tokenization). + +This example fine-tunes GPT-2 on the WikiText-2 dataset. The loss function is a causal language modeling loss (perplexity). + +.. code-block:: bash + export WIKITEXT_2_DATASET=/path/to/wikitext_dataset + + python run_lm_finetuning.py + --output_dir=output + --model_type=gpt2 + --model_name_or_path=gpt2 + --do_train + --train_data_file=$WIKITEXT_2_DATASET/wiki.train.raw + --do_eval + --eval_data_file=$WIKITEXT_2_DATASET/wiki.test.raw + +This takes about half an hour to train on a single K80 GPU and about one minute for the evaluation to run. +It reaches a score of about 20 perplexity once fine-tuned on the dataset. + +This example fine-tunes RoBERTa on the WikiText-2 dataset. The loss function is a masked language modeling loss (masked perplexity). +The `--mlm` flag is necessary to fine-tune BERT/RoBERTa on masked language modeling. + +.. code-block:: bash + export WIKITEXT_2_DATASET=/path/to/wikitext_dataset + + python run_lm_finetuning.py + --output_dir=output + --model_type=roberta + --model_name_or_path=roberta-base + --do_train + --train_data_file=$WIKITEXT_2_DATASET/wiki.train.raw + --do_eval + --eval_data_file=$WIKITEXT_2_DATASET/wiki.test.raw + --mlm .. _fine-tuning-BERT-large: