From e80be7f1d0836d352f2777a8f971dcbae2cd8882 Mon Sep 17 00:00:00 2001 From: Stefan Schweter Date: Fri, 1 May 2020 17:06:58 +0200 Subject: [PATCH] docs: add xlm-roberta section to multi-lingual section (#4101) --- docs/source/multilingual.rst | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/source/multilingual.rst b/docs/source/multilingual.rst index 78122296..dc68a69b 100644 --- a/docs/source/multilingual.rst +++ b/docs/source/multilingual.rst @@ -104,4 +104,16 @@ BERT has two checkpoints that can be used for multi-lingual tasks: - ``bert-base-multilingual-cased`` (Masked language modeling + Next sentence prediction, 104 languages) These checkpoints do not require language embeddings at inference time. They should identify the language -used in the context and infer accordingly. \ No newline at end of file +used in the context and infer accordingly. + +XLM-RoBERTa +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +XLM-RoBERTa was trained on 2.5TB of newly created clean CommonCrawl data in 100 languages. It provides strong +gains over previously released multi-lingual models like mBERT or XLM on downstream taks like classification, +sequence labeling and question answering. + +Two XLM-RoBERTa checkpoints can be used for multi-lingual tasks: + +- ``xlm-roberta-base`` (Masked language modeling, 100 languages) +- ``xlm-roberta-large`` (Masked language modeling, 100 languages)