Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 thomwolf
|
55c09d5915 | ||
|
thomwolf
|
25f47b15c0 | ||
|
Morgan Funtowicz
|
b92dda1e98 | ||
|
Morgan Funtowicz
|
144a0ef138 | ||
|
Morgan Funtowicz
|
78fab4cc85 |
+2
-3
@@ -3,7 +3,7 @@ cd docs
|
||||
function deploy_doc(){
|
||||
echo "Creating doc at commit $1 and pushing to folder $2"
|
||||
git checkout $1
|
||||
if [ ! -z "$2" ]
|
||||
if [ ! -z "$2" ]
|
||||
then
|
||||
if [ -d "$dir/$2" ]; then
|
||||
echo "Directory" $2 "already exists"
|
||||
@@ -17,7 +17,7 @@ function deploy_doc(){
|
||||
fi
|
||||
}
|
||||
|
||||
deploy_doc "master"
|
||||
deploy_doc "master"
|
||||
deploy_doc "b33a385" v1.0.0
|
||||
deploy_doc "fe02e45" v1.1.0
|
||||
deploy_doc "89fd345" v1.2.0
|
||||
@@ -25,4 +25,3 @@ deploy_doc "fc9faa8" v2.0.0
|
||||
deploy_doc "3ddce1d" v2.1.1
|
||||
deploy_doc "3616209" v2.2.0
|
||||
deploy_doc "d0f8b9a" v2.3.0
|
||||
deploy_doc "6664ea9" v2.4.0
|
||||
@@ -1,17 +1,17 @@
|
||||
---
|
||||
name: "\U0001F5A5 New benchmark"
|
||||
about: Benchmark a part of this library and share your results
|
||||
name: "\U0001F5A5 New Benchmark"
|
||||
about: You benchmark a part of this library and would like to share your results
|
||||
title: "[Benchmark]"
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
# 🖥 Benchmarking `transformers`
|
||||
# Benchmarking Transformers
|
||||
|
||||
## Benchmark
|
||||
|
||||
Which part of `transformers` did you benchmark?
|
||||
Which part of Transformers did you benchmark?
|
||||
|
||||
## Set-up
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
name: "\U0001F31F New model addition"
|
||||
name: "\U0001F31FNew model addition"
|
||||
about: Submit a proposal/request to implement a new Transformer-based model
|
||||
title: ''
|
||||
labels: ''
|
||||
@@ -7,14 +7,18 @@ assignees: ''
|
||||
|
||||
---
|
||||
|
||||
# 🌟 New model addition
|
||||
# 🌟New model addition
|
||||
|
||||
## Model description
|
||||
|
||||
<!-- Important information -->
|
||||
|
||||
## Open source status
|
||||
## Open Source status
|
||||
|
||||
* [ ] the model implementation is available: (give details)
|
||||
* [ ] the model weights are available: (give details)
|
||||
* [ ] who are the authors: (mention them, if possible by @gh-username)
|
||||
* [ ] who are the authors: (mention them)
|
||||
|
||||
## Additional context
|
||||
|
||||
<!-- Add any other context about the problem here. -->
|
||||
|
||||
@@ -1,29 +1,29 @@
|
||||
---
|
||||
name: "\U0001F41B Bug Report"
|
||||
about: Submit a bug report to help us improve transformers
|
||||
about: Submit a bug report to help us improve PyTorch Transformers
|
||||
title: ''
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
# 🐛 Bug
|
||||
## 🐛 Bug
|
||||
|
||||
## Information
|
||||
<!-- Important information -->
|
||||
|
||||
Model I am using (Bert, XLNet ...):
|
||||
Model I am using (Bert, XLNet....):
|
||||
|
||||
Language I am using the model on (English, Chinese ...):
|
||||
Language I am using the model on (English, Chinese....):
|
||||
|
||||
The problem arises when using:
|
||||
* [ ] the official example scripts: (give details below)
|
||||
* [ ] my own modified scripts: (give details below)
|
||||
The problem arise when using:
|
||||
* [ ] the official example scripts: (give details)
|
||||
* [ ] my own modified scripts: (give details)
|
||||
|
||||
The tasks I am working on is:
|
||||
* [ ] an official GLUE/SQUaD task: (give the name)
|
||||
* [ ] my own task or dataset: (give details below)
|
||||
* [ ] my own task or dataset: (give details)
|
||||
|
||||
## To reproduce
|
||||
## To Reproduce
|
||||
|
||||
Steps to reproduce the behavior:
|
||||
|
||||
@@ -31,22 +31,22 @@ Steps to reproduce the behavior:
|
||||
2.
|
||||
3.
|
||||
|
||||
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
|
||||
Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
|
||||
Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
|
||||
<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
|
||||
|
||||
## Expected behavior
|
||||
|
||||
<!-- A clear and concise description of what you would expect to happen. -->
|
||||
<!-- A clear and concise description of what you expected to happen. -->
|
||||
|
||||
## Environment info
|
||||
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
|
||||
Don't forget to fill out the missing fields in that output! -->
|
||||
|
||||
- `transformers` version:
|
||||
- Platform:
|
||||
- Python version:
|
||||
- PyTorch version (GPU?):
|
||||
- Tensorflow version (GPU?):
|
||||
- Using GPU in script?:
|
||||
- Using distributed or parallel set-up in script?:
|
||||
## Environment
|
||||
|
||||
* OS:
|
||||
* Python version:
|
||||
* PyTorch version:
|
||||
* PyTorch Transformers version (or branch):
|
||||
* Using GPU ?
|
||||
* Distributed or parallel setup ?
|
||||
* Any other relevant information:
|
||||
|
||||
## Additional context
|
||||
|
||||
<!-- Add any other context about the problem here. -->
|
||||
|
||||
@@ -1,25 +1,20 @@
|
||||
---
|
||||
name: "\U0001F680 Feature request"
|
||||
about: Submit a proposal/request for a new transformers feature
|
||||
name: "\U0001F680 Feature Request"
|
||||
about: Submit a proposal/request for a new PyTorch Transformers feature
|
||||
title: ''
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
# 🚀 Feature request
|
||||
## 🚀 Feature
|
||||
|
||||
<!-- A clear and concise description of the feature proposal.
|
||||
Please provide a link to the paper and code in case they exist. -->
|
||||
<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
|
||||
|
||||
## Motivation
|
||||
|
||||
<!-- Please outline the motivation for the proposal. Is your feature request
|
||||
related to a problem? e.g., I'm always frustrated when [...]. If this is related
|
||||
to another GitHub issue, please link here too. -->
|
||||
<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->
|
||||
|
||||
## Your contribution
|
||||
## Additional context
|
||||
|
||||
<!-- Is there any way that you could help, e.g. by submitting a PR?
|
||||
Make sure to read the CONTRIBUTING.MD readme:
|
||||
https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->
|
||||
<!-- Add any other context or screenshots about the feature request here. -->
|
||||
|
||||
@@ -1,57 +1,47 @@
|
||||
---
|
||||
name: "\U0001F4DA Migration from pytorch-pretrained-bert or pytorch-transformers"
|
||||
about: Report a problem when migrating from pytorch-pretrained-bert or pytorch-transformers to transformers
|
||||
name: "\U0001F4DA Migration from PyTorch-pretrained-Bert"
|
||||
about: Report a problem when migrating from PyTorch-pretrained-Bert to Transformers
|
||||
title: ''
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
# 📚 Migration
|
||||
|
||||
## Information
|
||||
## 📚 Migration
|
||||
|
||||
<!-- Important information -->
|
||||
|
||||
Model I am using (Bert, XLNet ...):
|
||||
Model I am using (Bert, XLNet....):
|
||||
|
||||
Language I am using the model on (English, Chinese ...):
|
||||
Language I am using the model on (English, Chinese....):
|
||||
|
||||
The problem arises when using:
|
||||
* [ ] the official example scripts: (give details below)
|
||||
* [ ] my own modified scripts: (give details below)
|
||||
The problem arise when using:
|
||||
* [ ] the official example scripts: (give details)
|
||||
* [ ] my own modified scripts: (give details)
|
||||
|
||||
The tasks I am working on is:
|
||||
* [ ] an official GLUE/SQUaD task: (give the name)
|
||||
* [ ] my own task or dataset: (give details below)
|
||||
* [ ] my own task or dataset: (give details)
|
||||
|
||||
## Details
|
||||
Details of the issue:
|
||||
|
||||
<!-- A clear and concise description of the migration issue.
|
||||
If you have code snippets, please provide it here as well.
|
||||
Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
|
||||
Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
|
||||
-->
|
||||
<!-- A clear and concise description of the migration issue. If you have code snippets, please provide it here as well. -->
|
||||
|
||||
## Environment info
|
||||
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
|
||||
Don't forget to fill out the missing fields in that output! -->
|
||||
|
||||
- `transformers` version:
|
||||
- Platform:
|
||||
- Python version:
|
||||
- PyTorch version (GPU?):
|
||||
- Tensorflow version (GPU?):
|
||||
- Using GPU in script?:
|
||||
- Using distributed or parallel set-up in script?:
|
||||
|
||||
<!-- IMPORTANT: which version of the former library do you use? -->
|
||||
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):
|
||||
## Environment
|
||||
|
||||
* OS:
|
||||
* Python version:
|
||||
* PyTorch version:
|
||||
* PyTorch Transformers version (or branch):
|
||||
* Using GPU ?
|
||||
* Distributed or parallel setup ?
|
||||
* Any other relevant information:
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] I have read the migration guide in the readme.
|
||||
([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
|
||||
[pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
|
||||
- [ ] I checked if a related official extension example runs on my machine.
|
||||
|
||||
## Additional context
|
||||
|
||||
<!-- Add any other context about the problem here. -->
|
||||
|
||||
@@ -1,29 +1,12 @@
|
||||
---
|
||||
name: "❓ Questions & Help"
|
||||
about: Post your general questions on Stack Overflow tagged huggingface-transformers
|
||||
name: "❓Questions & Help"
|
||||
about: Start a general discussion related to PyTorch Transformers
|
||||
title: ''
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
# ❓ Questions & Help
|
||||
## ❓ Questions & Help
|
||||
|
||||
<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
|
||||
new models and benchmarks, and migration questions. For all other questions,
|
||||
we direct you to Stack Overflow (SO) where a whole community of PyTorch and
|
||||
Tensorflow enthusiast can help you out. Make sure to tag your question with the
|
||||
right deep learning framework as well as the huggingface-transformers tag:
|
||||
https://stackoverflow.com/questions/tagged/huggingface-transformers
|
||||
|
||||
If your question wasn't answered after a period of time on Stack Overflow, you
|
||||
can always open a question on GitHub. You should then link to the SO question
|
||||
that you posted.
|
||||
-->
|
||||
|
||||
## Details
|
||||
<!-- Description of your issue -->
|
||||
|
||||
<!-- You should first ask your question on SO, and only if
|
||||
you didn't get an answer ask it here on GitHub. -->
|
||||
**A link to original question on Stack Overflow**:
|
||||
<!-- A clear and concise description of the question. -->
|
||||
|
||||
+7
-3
@@ -41,10 +41,14 @@ Did not find it? :( So we can act quickly on it, please follow these steps:
|
||||
less than 30s;
|
||||
* Provide the *full* traceback if an exception is raised.
|
||||
|
||||
To get the OS and software versions automatically, you can run the following command:
|
||||
To get the OS and software versions, execute the following code and copy-paste
|
||||
the output:
|
||||
|
||||
```bash
|
||||
python transformers-cli env
|
||||
```
|
||||
import platform; print("Platform", platform.platform())
|
||||
import sys; print("Python", sys.version)
|
||||
import torch; print("PyTorch", torch.__version__)
|
||||
import tensorflow; print("Tensorflow", tensorflow.__version__)
|
||||
```
|
||||
|
||||
### Do you want to implement a new model?
|
||||
|
||||
@@ -60,7 +60,7 @@ Choose the right framework for every part of a model's lifetime
|
||||
| [Quick tour: Share your models ](#Quick-tour-of-model-sharing) | Upload and share your fine-tuned models with the community |
|
||||
| [Migrating from pytorch-transformers to transformers](#Migrating-from-pytorch-transformers-to-transformers) | Migrating your code from pytorch-transformers to transformers |
|
||||
| [Migrating from pytorch-pretrained-bert to pytorch-transformers](#Migrating-from-pytorch-pretrained-bert-to-transformers) | Migrating your code from pytorch-pretrained-bert to transformers |
|
||||
| [Documentation][(v2.4.0)](https://huggingface.co/transformers/v2.4.0)[(v2.3.0)](https://huggingface.co/transformers/v2.3.0)[(v2.2.0/v2.2.1/v2.2.2)](https://huggingface.co/transformers/v2.2.0) [(v2.1.1)](https://huggingface.co/transformers/v2.1.1) [(v2.0.0)](https://huggingface.co/transformers/v2.0.0) [(v1.2.0)](https://huggingface.co/transformers/v1.2.0) [(v1.1.0)](https://huggingface.co/transformers/v1.1.0) [(v1.0.0)](https://huggingface.co/transformers/v1.0.0) [(master)](https://huggingface.co/transformers) | Full API documentation and more |
|
||||
| [Documentation][(v2.3.0)](https://huggingface.co/transformers/v2.3.0)[(v2.2.0/v2.2.1/v2.2.2)](https://huggingface.co/transformers/v2.2.0) [(v2.1.1)](https://huggingface.co/transformers/v2.1.1) [(v2.0.0)](https://huggingface.co/transformers/v2.0.0) [(v1.2.0)](https://huggingface.co/transformers/v1.2.0) [(v1.1.0)](https://huggingface.co/transformers/v1.1.0) [(v1.0.0)](https://huggingface.co/transformers/v1.0.0) [(master)](https://huggingface.co/transformers) | Full API documentation and more |
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -160,9 +160,8 @@ At some point in the future, you'll be able to seamlessly move from pre-training
|
||||
12. **[T5](https://github.com/google-research/text-to-text-transfer-transformer)** (from Google AI) released with the paper [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683) by Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.
|
||||
13. **[XLM-RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/xlmr)** (from Facebook AI), released together with the paper [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.
|
||||
14. **[MMBT](https://github.com/facebookresearch/mmbt/)** (from Facebook), released together with the paper a [Supervised Multimodal Bitransformers for Classifying Images and Text](https://arxiv.org/pdf/1909.02950.pdf) by Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine.
|
||||
15. **[FlauBERT](https://github.com/getalp/Flaubert)** (from CNRS) released with the paper [FlauBERT: Unsupervised Language Model Pre-training for French](https://arxiv.org/abs/1912.05372) by Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, Didier Schwab.
|
||||
16. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users).
|
||||
17. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.
|
||||
15. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users).
|
||||
16. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.
|
||||
|
||||
These implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations (e.g. ~93 F1 on SQuAD for BERT Whole-Word-Masking, ~88 F1 on RocStories for OpenAI GPT, ~18.3 perplexity on WikiText 103 for Transformer-XL, ~0.916 Peason R coefficient on STS-B for XLNet). You can find more details on the performances in the Examples section of the [documentation](https://huggingface.co/transformers/examples.html).
|
||||
|
||||
@@ -521,9 +520,8 @@ You can create `Pipeline` objects for the following down-stream tasks:
|
||||
- `feature-extraction`: Generates a tensor representation for the input sequence
|
||||
- `ner`: Generates named entity mapping for each word in the input sequence.
|
||||
- `sentiment-analysis`: Gives the polarity (positive / negative) of the whole input sequence.
|
||||
- `text-classification`: Initialize a `TextClassificationPipeline` directly, or see `sentiment-analysis` for an example.
|
||||
- `question-answering`: Provided some context and a question refering to the context, it will extract the answer to the question in the context.
|
||||
- `fill-mask`: Takes an input sequence containing a masked token (e.g. `<mask>`) and return list of most probable filled sequences, with their probabilities.
|
||||
- `question-answering`: Provided some context and a question refering to the context, it will extract the answer to the question
|
||||
in the context.
|
||||
|
||||
```python
|
||||
from transformers import pipeline
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ author = u'huggingface'
|
||||
# The short X.Y version
|
||||
version = u''
|
||||
# The full version, including alpha/beta/rc tags
|
||||
release = u'2.4.1'
|
||||
release = u'2.3.0'
|
||||
|
||||
|
||||
# -- General configuration ---------------------------------------------------
|
||||
|
||||
@@ -51,7 +51,6 @@ The library currently contains PyTorch and Tensorflow implementations, pre-train
|
||||
10. `CamemBERT <https://huggingface.co/transformers/model_doc/camembert.html>`_ (from FAIR, Inria, Sorbonne Université) released together with the paper `CamemBERT: a Tasty French Language Model <https://arxiv.org/abs/1911.03894>`_ by Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suarez, Yoann Dupont, Laurent Romary, Eric Villemonte de la Clergerie, Djame Seddah, and Benoît Sagot.
|
||||
11. `ALBERT <https://github.com/google-research/ALBERT>`_ (from Google Research), released together with the paper a `ALBERT: A Lite BERT for Self-supervised Learning of Language Representations <https://arxiv.org/abs/1909.11942>`_ by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.
|
||||
12. `XLM-RoBERTa <https://github.com/pytorch/fairseq/tree/master/examples/xlmr>`_ (from Facebook AI), released together with the paper `Unsupervised Cross-lingual Representation Learning at Scale <https://arxiv.org/abs/1911.02116>`_ by Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.
|
||||
13. `FlauBERT <https://github.com/getalp/Flaubert>`_ (from CNRS) released with the paper `FlauBERT: Unsupervised Language Model Pre-training for French <https://arxiv.org/abs/1912.05372>`_ by Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, Didier Schwab.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
@@ -98,5 +97,4 @@ The library currently contains PyTorch and Tensorflow implementations, pre-train
|
||||
model_doc/ctrl
|
||||
model_doc/camembert
|
||||
model_doc/albert
|
||||
model_doc/xlmroberta
|
||||
model_doc/flaubert
|
||||
model_doc/xlmroberta
|
||||
@@ -29,13 +29,6 @@ Instantiating one of ``AutoModel``, ``AutoConfig`` and ``AutoTokenizer`` will di
|
||||
:members:
|
||||
|
||||
|
||||
``AutoModelForPreTraining``
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.AutoModelForPreTraining
|
||||
:members:
|
||||
|
||||
|
||||
``AutoModelWithLMHead``
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
|
||||
@@ -69,31 +69,3 @@ CamembertForTokenClassification
|
||||
|
||||
.. autoclass:: transformers.CamembertForTokenClassification
|
||||
:members:
|
||||
|
||||
|
||||
TFCamembertModel
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TFCamembertModel
|
||||
:members:
|
||||
|
||||
|
||||
TFCamembertForMaskedLM
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TFCamembertForMaskedLM
|
||||
:members:
|
||||
|
||||
|
||||
TFCamembertForSequenceClassification
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TFCamembertForSequenceClassification
|
||||
:members:
|
||||
|
||||
|
||||
TFCamembertForTokenClassification
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TFCamembertForTokenClassification
|
||||
:members:
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
FlauBERT
|
||||
----------------------------------------------------
|
||||
|
||||
The FlauBERT model was proposed in the paper
|
||||
`FlauBERT: Unsupervised Language Model Pre-training for French <https://arxiv.org/abs/1912.05372>`__ by Hang Le et al.
|
||||
It's a transformer pre-trained using a masked language modeling (MLM) objective (BERT-like).
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Language models have become a key step to achieve state-of-the art results in many different Natural Language
|
||||
Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient
|
||||
way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their
|
||||
contextualization at the sentence level. This has been widely demonstrated for English using contextualized
|
||||
representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et
|
||||
al., 2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large
|
||||
and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre
|
||||
for Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text
|
||||
classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most
|
||||
of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified
|
||||
evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared
|
||||
to the research community for further reproducible experiments in French NLP.*
|
||||
|
||||
|
||||
FlaubertConfig
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.FlaubertConfig
|
||||
:members:
|
||||
|
||||
|
||||
FlaubertTokenizer
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.FlaubertTokenizer
|
||||
:members:
|
||||
|
||||
|
||||
FlaubertModel
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.FlaubertModel
|
||||
:members:
|
||||
|
||||
|
||||
FlaubertWithLMHeadModel
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.FlaubertWithLMHeadModel
|
||||
:members:
|
||||
|
||||
|
||||
FlaubertForSequenceClassification
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.FlaubertForSequenceClassification
|
||||
:members:
|
||||
|
||||
|
||||
FlaubertForQuestionAnsweringSimple
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.FlaubertForQuestionAnsweringSimple
|
||||
:members:
|
||||
|
||||
|
||||
FlaubertForQuestionAnswering
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.FlaubertForQuestionAnswering
|
||||
:members:
|
||||
|
||||
|
||||
@@ -73,30 +73,3 @@ XLMRobertaForTokenClassification
|
||||
.. autoclass:: transformers.XLMRobertaForTokenClassification
|
||||
:members:
|
||||
|
||||
|
||||
TFXLMRobertaModel
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TFXLMRobertaModel
|
||||
:members:
|
||||
|
||||
|
||||
TFXLMRobertaForMaskedLM
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TFXLMRobertaForMaskedLM
|
||||
:members:
|
||||
|
||||
|
||||
TFXLMRobertaForSequenceClassification
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TFXLMRobertaForSequenceClassification
|
||||
:members:
|
||||
|
||||
|
||||
TFXLMRobertaForTokenClassification
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TFXLMRobertaForTokenClassification
|
||||
:members:
|
||||
|
||||
@@ -88,10 +88,6 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
|
||||
| | ``bert-base-finnish-uncased-v1`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
||||
| | | | Trained on uncased Finnish text. |
|
||||
| | | (see `details on turkunlp.org <http://turkunlp.org/FinBERT/>`__). |
|
||||
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| | ``bert-base-dutch-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
||||
| | | | Trained on cased Dutch text. |
|
||||
| | | (see `details on wietsedv repository <https://github.com/wietsedv/bertje/>`__). |
|
||||
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| GPT | ``openai-gpt`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
||||
| | | | OpenAI GPT English model |
|
||||
@@ -251,22 +247,6 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
|
||||
| | ``xlm-roberta-large`` | | ~355M parameters with 24-layers, 1027-hidden-state, 4096 feed-forward hidden-state, 16-heads, |
|
||||
| | | | Trained on 2.5 TB of newly created clean CommonCrawl data in 100 languages |
|
||||
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| FlauBERT | ``flaubert-small-cased`` | | 6-layer, 512-hidden, 8-heads, 54M parameters |
|
||||
| | | | FlauBERT small architecture |
|
||||
| | | (see `details <https://github.com/getalp/Flaubert>`__) |
|
||||
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| | ``flaubert-base-uncased`` | | 12-layer, 768-hidden, 12-heads, 137M parameters |
|
||||
| | | | FlauBERT base architecture with uncased vocabulary |
|
||||
| | | (see `details <https://github.com/getalp/Flaubert>`__) |
|
||||
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| | ``flaubert-base-cased`` | | 12-layer, 768-hidden, 12-heads, 138M parameters |
|
||||
| | | | FlauBERT base architecture with cased vocabulary |
|
||||
| | | (see `details <https://github.com/getalp/Flaubert>`__) |
|
||||
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| | ``flaubert-large-cased`` | | 24-layer, 1024-hidden, 16-heads, 373M parameters |
|
||||
| | | | FlauBERT large architecture |
|
||||
| | | (see `details <https://github.com/getalp/Flaubert>`__) |
|
||||
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
||||
|
||||
|
||||
.. <https://huggingface.co/transformers/examples.html>`__
|
||||
|
||||
@@ -299,8 +299,8 @@ model = Model2Model.from_pretrained('fine-tuned-weights')
|
||||
model.eval()
|
||||
|
||||
# If you have a GPU, put everything on cuda
|
||||
question_tensor = question_tensor.to('cuda')
|
||||
answer_tensor = answer_tensor.to('cuda')
|
||||
question_tensor = encoded_question.to('cuda')
|
||||
answer_tensor = encoded_answer.to('cuda')
|
||||
model.to('cuda')
|
||||
|
||||
# Predict all tokens
|
||||
|
||||
+6
-6
@@ -404,12 +404,12 @@ exact_match = 81.22
|
||||
#### Distributed training
|
||||
|
||||
|
||||
Here is an example using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD1.1:
|
||||
Here is an example using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD1.0:
|
||||
|
||||
```bash
|
||||
python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
|
||||
python -m torch.distributed.launch --nproc_per_node=8 run_squad.py \
|
||||
--model_type bert \
|
||||
--model_name_or_path bert-large-uncased-whole-word-masking \
|
||||
--model_name_or_path bert-base-cased \
|
||||
--do_train \
|
||||
--do_eval \
|
||||
--do_lower_case \
|
||||
@@ -419,9 +419,9 @@ python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
|
||||
--num_train_epochs 2 \
|
||||
--max_seq_length 384 \
|
||||
--doc_stride 128 \
|
||||
--output_dir ./examples/models/wwm_uncased_finetuned_squad/ \
|
||||
--per_gpu_eval_batch_size=3 \
|
||||
--per_gpu_train_batch_size=3 \
|
||||
--output_dir ../models/wwm_uncased_finetuned_squad/ \
|
||||
--per_gpu_train_batch_size 24 \
|
||||
--gradient_accumulation_steps 12
|
||||
```
|
||||
|
||||
Training with the previously defined hyper-parameters yields the following results:
|
||||
|
||||
@@ -221,7 +221,6 @@ def main():
|
||||
top_k=args.k,
|
||||
top_p=args.p,
|
||||
repetition_penalty=args.repetition_penalty,
|
||||
do_sample=True,
|
||||
)
|
||||
|
||||
# Batch size == 1. to add more examples please use num_return_sequences > 1
|
||||
|
||||
@@ -41,9 +41,6 @@ from transformers import (
|
||||
DistilBertConfig,
|
||||
DistilBertForSequenceClassification,
|
||||
DistilBertTokenizer,
|
||||
FlaubertConfig,
|
||||
FlaubertForSequenceClassification,
|
||||
FlaubertTokenizer,
|
||||
RobertaConfig,
|
||||
RobertaForSequenceClassification,
|
||||
RobertaTokenizer,
|
||||
@@ -83,7 +80,6 @@ ALL_MODELS = sum(
|
||||
DistilBertConfig,
|
||||
AlbertConfig,
|
||||
XLMRobertaConfig,
|
||||
FlaubertConfig,
|
||||
)
|
||||
),
|
||||
(),
|
||||
@@ -97,7 +93,6 @@ MODEL_CLASSES = {
|
||||
"distilbert": (DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer),
|
||||
"albert": (AlbertConfig, AlbertForSequenceClassification, AlbertTokenizer),
|
||||
"xlmroberta": (XLMRobertaConfig, XLMRobertaForSequenceClassification, XLMRobertaTokenizer),
|
||||
"flaubert": (FlaubertConfig, FlaubertForSequenceClassification, FlaubertTokenizer),
|
||||
}
|
||||
|
||||
|
||||
@@ -485,7 +480,7 @@ def main():
|
||||
parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
|
||||
parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
|
||||
parser.add_argument(
|
||||
"--evaluate_during_training", action="store_true", help="Run evaluation during training at each logging step.",
|
||||
"--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model.",
|
||||
@@ -518,8 +513,8 @@ def main():
|
||||
)
|
||||
parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
|
||||
|
||||
parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument(
|
||||
"--eval_all_checkpoints",
|
||||
action="store_true",
|
||||
|
||||
@@ -195,7 +195,6 @@ def _rotate_checkpoints(args, checkpoint_prefix="checkpoint", use_mtime=False) -
|
||||
|
||||
def mask_tokens(inputs: torch.Tensor, tokenizer: PreTrainedTokenizer, args) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||
""" Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. """
|
||||
inputs = inputs.clone().type(dtype=torch.long)
|
||||
labels = inputs.clone()
|
||||
# We sample a few tokens in each sequence for masked-LM training (with probability args.mlm_probability defaults to 0.15 in Bert/RoBERTa)
|
||||
probability_matrix = torch.full(labels.shape, args.mlm_probability)
|
||||
@@ -203,9 +202,8 @@ def mask_tokens(inputs: torch.Tensor, tokenizer: PreTrainedTokenizer, args) -> T
|
||||
tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()
|
||||
]
|
||||
probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
|
||||
if tokenizer._pad_token is not None:
|
||||
padding_mask = labels.eq(tokenizer.pad_token_id)
|
||||
probability_matrix.masked_fill_(padding_mask, value=0.0)
|
||||
padding_mask = labels.eq(tokenizer.pad_token_id)
|
||||
probability_matrix.masked_fill_(padding_mask, value=0.0)
|
||||
masked_indices = torch.bernoulli(probability_matrix).bool()
|
||||
labels[~masked_indices] = -100 # We only compute loss on masked tokens
|
||||
|
||||
@@ -230,8 +228,6 @@ def train(args, train_dataset, model: PreTrainedModel, tokenizer: PreTrainedToke
|
||||
args.train_batch_size = args.per_gpu_train_batch_size * max(1, args.n_gpu)
|
||||
|
||||
def collate(examples: List[torch.Tensor]):
|
||||
if tokenizer._pad_token is None:
|
||||
return pad_sequence(examples, batch_first=True)
|
||||
return pad_sequence(examples, batch_first=True, padding_value=tokenizer.pad_token_id)
|
||||
|
||||
train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
|
||||
@@ -425,8 +421,6 @@ def evaluate(args, model: PreTrainedModel, tokenizer: PreTrainedTokenizer, prefi
|
||||
# Note that DistributedSampler samples randomly
|
||||
|
||||
def collate(examples: List[torch.Tensor]):
|
||||
if tokenizer._pad_token is None:
|
||||
return pad_sequence(examples, batch_first=True)
|
||||
return pad_sequence(examples, batch_first=True, padding_value=tokenizer.pad_token_id)
|
||||
|
||||
eval_sampler = SequentialSampler(eval_dataset)
|
||||
@@ -575,8 +569,8 @@ def main():
|
||||
)
|
||||
parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
|
||||
|
||||
parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument(
|
||||
"--save_total_limit",
|
||||
type=int,
|
||||
|
||||
@@ -478,8 +478,8 @@ def main():
|
||||
)
|
||||
parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
|
||||
|
||||
parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument(
|
||||
"--eval_all_checkpoints",
|
||||
action="store_true",
|
||||
|
||||
+2
-2
@@ -485,8 +485,8 @@ def main():
|
||||
)
|
||||
parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
|
||||
|
||||
parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument(
|
||||
"--eval_all_checkpoints",
|
||||
action="store_true",
|
||||
|
||||
+2
-18
@@ -219,11 +219,6 @@ def train(args, train_dataset, model, tokenizer):
|
||||
inputs.update({"cls_index": batch[5], "p_mask": batch[6]})
|
||||
if args.version_2_with_negative:
|
||||
inputs.update({"is_impossible": batch[7]})
|
||||
if hasattr(model, "config") and hasattr(model.config, "lang2id"):
|
||||
inputs.update(
|
||||
{"langs": (torch.ones(batch[0].shape, dtype=torch.int64) * args.lang_id).to(args.device)}
|
||||
)
|
||||
|
||||
outputs = model(**inputs)
|
||||
# model outputs are always tuple in transformers (see doc)
|
||||
loss = outputs[0]
|
||||
@@ -335,11 +330,6 @@ def evaluate(args, model, tokenizer, prefix=""):
|
||||
# XLNet and XLM use more arguments for their predictions
|
||||
if args.model_type in ["xlnet", "xlm"]:
|
||||
inputs.update({"cls_index": batch[4], "p_mask": batch[5]})
|
||||
# for lang_id-sensitive xlm models
|
||||
if hasattr(model, "config") and hasattr(model.config, "lang2id"):
|
||||
inputs.update(
|
||||
{"langs": (torch.ones(batch[0].shape, dtype=torch.int64) * args.lang_id).to(args.device)}
|
||||
)
|
||||
|
||||
outputs = model(**inputs)
|
||||
|
||||
@@ -645,15 +635,9 @@ def main():
|
||||
help="If true, all of the warnings related to data processing will be printed. "
|
||||
"A number of warnings are expected for a normal SQuAD evaluation.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lang_id",
|
||||
default=0,
|
||||
type=int,
|
||||
help="language id of input for language-specific xlm models (see tokenization_xlm.PRETRAINED_INIT_CONFIGURATION)",
|
||||
)
|
||||
|
||||
parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument(
|
||||
"--eval_all_checkpoints",
|
||||
action="store_true",
|
||||
|
||||
@@ -99,6 +99,9 @@ if TASK == "mrpc":
|
||||
inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors="pt")
|
||||
inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors="pt")
|
||||
|
||||
del inputs_1["special_tokens_mask"]
|
||||
del inputs_2["special_tokens_mask"]
|
||||
|
||||
pred_1 = pytorch_model(**inputs_1)[0].argmax().item()
|
||||
pred_2 = pytorch_model(**inputs_2)[0].argmax().item()
|
||||
print("sentence_1 is", "a paraphrase" if pred_1 else "not a paraphrase", "of sentence_0")
|
||||
|
||||
@@ -473,8 +473,8 @@ def main():
|
||||
)
|
||||
parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
|
||||
|
||||
parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
|
||||
parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
|
||||
parser.add_argument(
|
||||
"--eval_all_checkpoints",
|
||||
action="store_true",
|
||||
|
||||
@@ -73,7 +73,7 @@ def read_examples_from_file(data_dir, mode):
|
||||
# Examples could have no label for mode = "test"
|
||||
labels.append("O")
|
||||
if words:
|
||||
examples.append(InputExample(guid="{}-{}".format(mode, guid_index), words=words, labels=labels))
|
||||
examples.append(InputExample(guid="%s-%d".format(mode, guid_index), words=words, labels=labels))
|
||||
return examples
|
||||
|
||||
|
||||
|
||||
@@ -1,117 +0,0 @@
|
||||
# Swedish BERT Models
|
||||
|
||||
The National Library of Sweden / KBLab releases three pretrained language models based on BERT and ALBERT. The models are trained on aproximately 15-20GB of text (200M sentences, 3000M tokens) from various sources (books, news, government publications, swedish wikipedia and internet forums) aiming to provide a representative BERT model for Swedish text. A more complete description will be published later on.
|
||||
|
||||
The following three models are currently available:
|
||||
|
||||
- **bert-base-swedish-cased** (*v1*) - A BERT trained with the same hyperparameters as first published by Google.
|
||||
- **bert-base-swedish-cased-ner** (*experimental*) - a BERT fine-tuned for NER using SUC 3.0.
|
||||
- **albert-base-swedish-cased-alpha** (*alpha*) - A first attempt at an ALBERT for Swedish.
|
||||
|
||||
All models are cased and trained with whole word masking.
|
||||
|
||||
## Files
|
||||
|
||||
| **name** | **files** |
|
||||
|---------------------------------|-----------|
|
||||
| bert-base-swedish-cased | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/vocab.txt), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/pytorch_model.bin) |
|
||||
| bert-base-swedish-cased-ner | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/vocab.txt) [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/pytorch_model.bin) |
|
||||
| albert-base-swedish-cased-alpha | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/config.json), [sentencepiece model](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/spiece.model), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/pytorch_model.bin) |
|
||||
|
||||
TensorFlow model weights will be released soon.
|
||||
|
||||
## Usage requirements / installation instructions
|
||||
|
||||
The examples below require Huggingface Transformers 2.4.1 and Pytorch 1.3.1 or greater. For Transformers<2.4.0 the tokenizer must be instantiated manually and the `do_lower_case` flag parameter set to `False` and `keep_accents` to `True` (for ALBERT).
|
||||
|
||||
To create an environment where the examples can be run, run the following in an terminal on your OS of choice.
|
||||
|
||||
```
|
||||
# git clone https://github.com/Kungbib/swedish-bert-models
|
||||
# cd swedish-bert-models
|
||||
# python3 -m venv venv
|
||||
# source venv/bin/activate
|
||||
# pip install --upgrade pip
|
||||
# pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### BERT Base Swedish
|
||||
|
||||
A standard BERT base for Swedish trained on a variety of sources. Vocabulary size is ~50k. Using Huggingface Transformers the model can be loaded in Python as follows:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel,AutoTokenizer
|
||||
|
||||
tok = AutoTokenizer.from_pretrained('KB/bert-base-swedish-cased')
|
||||
model = AutoModel.from_pretrained('KB/bert-base-swedish-cased')
|
||||
```
|
||||
|
||||
|
||||
### BERT base fine-tuned for Swedish NER
|
||||
|
||||
This model is fine-tuned on the SUC 3.0 dataset. Using the Huggingface pipeline the model can be easily instantiated. For Transformer<2.4.1 it seems the tokenizer must be loaded separately to disable lower-casing of input strings:
|
||||
|
||||
```python
|
||||
from transformers import pipeline
|
||||
|
||||
nlp = pipeline('ner', model='KB/bert-base-swedish-cased-ner', tokenizer='KB/bert-base-swedish-cased-ner')
|
||||
|
||||
nlp('Idag släpper KB tre språkmodeller.')
|
||||
```
|
||||
|
||||
Running the Python code above should produce in something like the result below. Entity types used are `TME` for time, `PRS` for personal names, `LOC` for locations, `EVN` for events and `ORG` for organisations. These labels are subject to change.
|
||||
|
||||
```python
|
||||
[ { 'word': 'Idag', 'score': 0.9998126029968262, 'entity': 'TME' },
|
||||
{ 'word': 'KB', 'score': 0.9814832210540771, 'entity': 'ORG' } ]
|
||||
```
|
||||
|
||||
The BERT tokenizer often splits words into multiple tokens, with the subparts starting with `##`, for example the string `Engelbert kör Volvo till Herrängens fotbollsklubb` gets tokenized as `Engel ##bert kör Volvo till Herr ##ängens fotbolls ##klubb`. To glue parts back together one can use something like this:
|
||||
|
||||
```python
|
||||
text = 'Engelbert tar Volvon till Tele2 Arena för att titta på Djurgården IF ' +\
|
||||
'som spelar fotboll i VM klockan två på kvällen.'
|
||||
|
||||
l = []
|
||||
for token in nlp(text):
|
||||
if token['word'].startswith('##'):
|
||||
l[-1]['word'] += token['word'][2:]
|
||||
else:
|
||||
l += [ token ]
|
||||
|
||||
print(l)
|
||||
```
|
||||
|
||||
Which should result in the following (though less cleanly formated):
|
||||
|
||||
```python
|
||||
[ { 'word': 'Engelbert', 'score': 0.99..., 'entity': 'PRS'},
|
||||
{ 'word': 'Volvon', 'score': 0.99..., 'entity': 'OBJ'},
|
||||
{ 'word': 'Tele2', 'score': 0.99..., 'entity': 'LOC'},
|
||||
{ 'word': 'Arena', 'score': 0.99..., 'entity': 'LOC'},
|
||||
{ 'word': 'Djurgården', 'score': 0.99..., 'entity': 'ORG'},
|
||||
{ 'word': 'IF', 'score': 0.99..., 'entity': 'ORG'},
|
||||
{ 'word': 'VM', 'score': 0.99..., 'entity': 'EVN'},
|
||||
{ 'word': 'klockan', 'score': 0.99..., 'entity': 'TME'},
|
||||
{ 'word': 'två', 'score': 0.99..., 'entity': 'TME'},
|
||||
{ 'word': 'på', 'score': 0.99..., 'entity': 'TME'},
|
||||
{ 'word': 'kvällen', 'score': 0.54..., 'entity': 'TME'} ]
|
||||
```
|
||||
|
||||
### ALBERT base
|
||||
|
||||
The easisest way to do this is, again, using Huggingface Transformers:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel,AutoTokenizer
|
||||
|
||||
tok = AutoTokenizer.from_pretrained('KB/albert-base-swedish-cased-alpha'),
|
||||
model = AutoModel.from_pretrained('KB/albert-base-swedish-cased-alpha')
|
||||
```
|
||||
|
||||
## Acknowledgements ❤️
|
||||
|
||||
- Resources from Stockholms University, Umeå University and Swedish Language Bank at Gothenburg University were used when fine-tuning BERT for NER.
|
||||
- Model pretraining was made partly in-house at the KBLab and partly (for material without active copyright) with the support of Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
|
||||
- Models are hosted on S3 by Huggingface 🤗
|
||||
|
||||
@@ -1,117 +0,0 @@
|
||||
# Swedish BERT Models
|
||||
|
||||
The National Library of Sweden / KBLab releases three pretrained language models based on BERT and ALBERT. The models are trained on aproximately 15-20GB of text (200M sentences, 3000M tokens) from various sources (books, news, government publications, swedish wikipedia and internet forums) aiming to provide a representative BERT model for Swedish text. A more complete description will be published later on.
|
||||
|
||||
The following three models are currently available:
|
||||
|
||||
- **bert-base-swedish-cased** (*v1*) - A BERT trained with the same hyperparameters as first published by Google.
|
||||
- **bert-base-swedish-cased-ner** (*experimental*) - a BERT fine-tuned for NER using SUC 3.0.
|
||||
- **albert-base-swedish-cased-alpha** (*alpha*) - A first attempt at an ALBERT for Swedish.
|
||||
|
||||
All models are cased and trained with whole word masking.
|
||||
|
||||
## Files
|
||||
|
||||
| **name** | **files** |
|
||||
|---------------------------------|-----------|
|
||||
| bert-base-swedish-cased | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/vocab.txt), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/pytorch_model.bin) |
|
||||
| bert-base-swedish-cased-ner | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/vocab.txt) [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/pytorch_model.bin) |
|
||||
| albert-base-swedish-cased-alpha | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/config.json), [sentencepiece model](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/spiece.model), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/pytorch_model.bin) |
|
||||
|
||||
TensorFlow model weights will be released soon.
|
||||
|
||||
## Usage requirements / installation instructions
|
||||
|
||||
The examples below require Huggingface Transformers 2.4.1 and Pytorch 1.3.1 or greater. For Transformers<2.4.0 the tokenizer must be instantiated manually and the `do_lower_case` flag parameter set to `False` and `keep_accents` to `True` (for ALBERT).
|
||||
|
||||
To create an environment where the examples can be run, run the following in an terminal on your OS of choice.
|
||||
|
||||
```
|
||||
# git clone https://github.com/Kungbib/swedish-bert-models
|
||||
# cd swedish-bert-models
|
||||
# python3 -m venv venv
|
||||
# source venv/bin/activate
|
||||
# pip install --upgrade pip
|
||||
# pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### BERT Base Swedish
|
||||
|
||||
A standard BERT base for Swedish trained on a variety of sources. Vocabulary size is ~50k. Using Huggingface Transformers the model can be loaded in Python as follows:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel,AutoTokenizer
|
||||
|
||||
tok = AutoTokenizer.from_pretrained('KB/bert-base-swedish-cased')
|
||||
model = AutoModel.from_pretrained('KB/bert-base-swedish-cased')
|
||||
```
|
||||
|
||||
|
||||
### BERT base fine-tuned for Swedish NER
|
||||
|
||||
This model is fine-tuned on the SUC 3.0 dataset. Using the Huggingface pipeline the model can be easily instantiated. For Transformer<2.4.1 it seems the tokenizer must be loaded separately to disable lower-casing of input strings:
|
||||
|
||||
```python
|
||||
from transformers import pipeline
|
||||
|
||||
nlp = pipeline('ner', model='KB/bert-base-swedish-cased-ner', tokenizer='KB/bert-base-swedish-cased-ner')
|
||||
|
||||
nlp('Idag släpper KB tre språkmodeller.')
|
||||
```
|
||||
|
||||
Running the Python code above should produce in something like the result below. Entity types used are `TME` for time, `PRS` for personal names, `LOC` for locations, `EVN` for events and `ORG` for organisations. These labels are subject to change.
|
||||
|
||||
```python
|
||||
[ { 'word': 'Idag', 'score': 0.9998126029968262, 'entity': 'TME' },
|
||||
{ 'word': 'KB', 'score': 0.9814832210540771, 'entity': 'ORG' } ]
|
||||
```
|
||||
|
||||
The BERT tokenizer often splits words into multiple tokens, with the subparts starting with `##`, for example the string `Engelbert kör Volvo till Herrängens fotbollsklubb` gets tokenized as `Engel ##bert kör Volvo till Herr ##ängens fotbolls ##klubb`. To glue parts back together one can use something like this:
|
||||
|
||||
```python
|
||||
text = 'Engelbert tar Volvon till Tele2 Arena för att titta på Djurgården IF ' +\
|
||||
'som spelar fotboll i VM klockan två på kvällen.'
|
||||
|
||||
l = []
|
||||
for token in nlp(text):
|
||||
if token['word'].startswith('##'):
|
||||
l[-1]['word'] += token['word'][2:]
|
||||
else:
|
||||
l += [ token ]
|
||||
|
||||
print(l)
|
||||
```
|
||||
|
||||
Which should result in the following (though less cleanly formated):
|
||||
|
||||
```python
|
||||
[ { 'word': 'Engelbert', 'score': 0.99..., 'entity': 'PRS'},
|
||||
{ 'word': 'Volvon', 'score': 0.99..., 'entity': 'OBJ'},
|
||||
{ 'word': 'Tele2', 'score': 0.99..., 'entity': 'LOC'},
|
||||
{ 'word': 'Arena', 'score': 0.99..., 'entity': 'LOC'},
|
||||
{ 'word': 'Djurgården', 'score': 0.99..., 'entity': 'ORG'},
|
||||
{ 'word': 'IF', 'score': 0.99..., 'entity': 'ORG'},
|
||||
{ 'word': 'VM', 'score': 0.99..., 'entity': 'EVN'},
|
||||
{ 'word': 'klockan', 'score': 0.99..., 'entity': 'TME'},
|
||||
{ 'word': 'två', 'score': 0.99..., 'entity': 'TME'},
|
||||
{ 'word': 'på', 'score': 0.99..., 'entity': 'TME'},
|
||||
{ 'word': 'kvällen', 'score': 0.54..., 'entity': 'TME'} ]
|
||||
```
|
||||
|
||||
### ALBERT base
|
||||
|
||||
The easisest way to do this is, again, using Huggingface Transformers:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel,AutoTokenizer
|
||||
|
||||
tok = AutoTokenizer.from_pretrained('KB/albert-base-swedish-cased-alpha'),
|
||||
model = AutoModel.from_pretrained('KB/albert-base-swedish-cased-alpha')
|
||||
```
|
||||
|
||||
## Acknowledgements ❤️
|
||||
|
||||
- Resources from Stockholms University, Umeå University and Swedish Language Bank at Gothenburg University were used when fine-tuning BERT for NER.
|
||||
- Model pretraining was made partly in-house at the KBLab and partly (for material without active copyright) with the support of Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
|
||||
- Models are hosted on S3 by Huggingface 🤗
|
||||
|
||||
@@ -1,117 +0,0 @@
|
||||
# Swedish BERT Models
|
||||
|
||||
The National Library of Sweden / KBLab releases three pretrained language models based on BERT and ALBERT. The models are trained on aproximately 15-20GB of text (200M sentences, 3000M tokens) from various sources (books, news, government publications, swedish wikipedia and internet forums) aiming to provide a representative BERT model for Swedish text. A more complete description will be published later on.
|
||||
|
||||
The following three models are currently available:
|
||||
|
||||
- **bert-base-swedish-cased** (*v1*) - A BERT trained with the same hyperparameters as first published by Google.
|
||||
- **bert-base-swedish-cased-ner** (*experimental*) - a BERT fine-tuned for NER using SUC 3.0.
|
||||
- **albert-base-swedish-cased-alpha** (*alpha*) - A first attempt at an ALBERT for Swedish.
|
||||
|
||||
All models are cased and trained with whole word masking.
|
||||
|
||||
## Files
|
||||
|
||||
| **name** | **files** |
|
||||
|---------------------------------|-----------|
|
||||
| bert-base-swedish-cased | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/vocab.txt), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased/pytorch_model.bin) |
|
||||
| bert-base-swedish-cased-ner | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/config.json), [vocab](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/vocab.txt) [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/bert-base-swedish-cased-ner/pytorch_model.bin) |
|
||||
| albert-base-swedish-cased-alpha | [config](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/config.json), [sentencepiece model](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/spiece.model), [pytorch_model.bin](https://s3.amazonaws.com/models.huggingface.co/bert/KB/albert-base-swedish-cased-alpha/pytorch_model.bin) |
|
||||
|
||||
TensorFlow model weights will be released soon.
|
||||
|
||||
## Usage requirements / installation instructions
|
||||
|
||||
The examples below require Huggingface Transformers 2.4.1 and Pytorch 1.3.1 or greater. For Transformers<2.4.0 the tokenizer must be instantiated manually and the `do_lower_case` flag parameter set to `False` and `keep_accents` to `True` (for ALBERT).
|
||||
|
||||
To create an environment where the examples can be run, run the following in an terminal on your OS of choice.
|
||||
|
||||
```
|
||||
# git clone https://github.com/Kungbib/swedish-bert-models
|
||||
# cd swedish-bert-models
|
||||
# python3 -m venv venv
|
||||
# source venv/bin/activate
|
||||
# pip install --upgrade pip
|
||||
# pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### BERT Base Swedish
|
||||
|
||||
A standard BERT base for Swedish trained on a variety of sources. Vocabulary size is ~50k. Using Huggingface Transformers the model can be loaded in Python as follows:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel,AutoTokenizer
|
||||
|
||||
tok = AutoTokenizer.from_pretrained('KB/bert-base-swedish-cased')
|
||||
model = AutoModel.from_pretrained('KB/bert-base-swedish-cased')
|
||||
```
|
||||
|
||||
|
||||
### BERT base fine-tuned for Swedish NER
|
||||
|
||||
This model is fine-tuned on the SUC 3.0 dataset. Using the Huggingface pipeline the model can be easily instantiated. For Transformer<2.4.1 it seems the tokenizer must be loaded separately to disable lower-casing of input strings:
|
||||
|
||||
```python
|
||||
from transformers import pipeline
|
||||
|
||||
nlp = pipeline('ner', model='KB/bert-base-swedish-cased-ner', tokenizer='KB/bert-base-swedish-cased-ner')
|
||||
|
||||
nlp('Idag släpper KB tre språkmodeller.')
|
||||
```
|
||||
|
||||
Running the Python code above should produce in something like the result below. Entity types used are `TME` for time, `PRS` for personal names, `LOC` for locations, `EVN` for events and `ORG` for organisations. These labels are subject to change.
|
||||
|
||||
```python
|
||||
[ { 'word': 'Idag', 'score': 0.9998126029968262, 'entity': 'TME' },
|
||||
{ 'word': 'KB', 'score': 0.9814832210540771, 'entity': 'ORG' } ]
|
||||
```
|
||||
|
||||
The BERT tokenizer often splits words into multiple tokens, with the subparts starting with `##`, for example the string `Engelbert kör Volvo till Herrängens fotbollsklubb` gets tokenized as `Engel ##bert kör Volvo till Herr ##ängens fotbolls ##klubb`. To glue parts back together one can use something like this:
|
||||
|
||||
```python
|
||||
text = 'Engelbert tar Volvon till Tele2 Arena för att titta på Djurgården IF ' +\
|
||||
'som spelar fotboll i VM klockan två på kvällen.'
|
||||
|
||||
l = []
|
||||
for token in nlp(text):
|
||||
if token['word'].startswith('##'):
|
||||
l[-1]['word'] += token['word'][2:]
|
||||
else:
|
||||
l += [ token ]
|
||||
|
||||
print(l)
|
||||
```
|
||||
|
||||
Which should result in the following (though less cleanly formated):
|
||||
|
||||
```python
|
||||
[ { 'word': 'Engelbert', 'score': 0.99..., 'entity': 'PRS'},
|
||||
{ 'word': 'Volvon', 'score': 0.99..., 'entity': 'OBJ'},
|
||||
{ 'word': 'Tele2', 'score': 0.99..., 'entity': 'LOC'},
|
||||
{ 'word': 'Arena', 'score': 0.99..., 'entity': 'LOC'},
|
||||
{ 'word': 'Djurgården', 'score': 0.99..., 'entity': 'ORG'},
|
||||
{ 'word': 'IF', 'score': 0.99..., 'entity': 'ORG'},
|
||||
{ 'word': 'VM', 'score': 0.99..., 'entity': 'EVN'},
|
||||
{ 'word': 'klockan', 'score': 0.99..., 'entity': 'TME'},
|
||||
{ 'word': 'två', 'score': 0.99..., 'entity': 'TME'},
|
||||
{ 'word': 'på', 'score': 0.99..., 'entity': 'TME'},
|
||||
{ 'word': 'kvällen', 'score': 0.54..., 'entity': 'TME'} ]
|
||||
```
|
||||
|
||||
### ALBERT base
|
||||
|
||||
The easisest way to do this is, again, using Huggingface Transformers:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel,AutoTokenizer
|
||||
|
||||
tok = AutoTokenizer.from_pretrained('KB/albert-base-swedish-cased-alpha'),
|
||||
model = AutoModel.from_pretrained('KB/albert-base-swedish-cased-alpha')
|
||||
```
|
||||
|
||||
## Acknowledgements ❤️
|
||||
|
||||
- Resources from Stockholms University, Umeå University and Swedish Language Bank at Gothenburg University were used when fine-tuning BERT for NER.
|
||||
- Model pretraining was made partly in-house at the KBLab and partly (for material without active copyright) with the support of Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
|
||||
- Models are hosted on S3 by Huggingface 🤗
|
||||
|
||||
@@ -1,114 +0,0 @@
|
||||
# UmBERTo Commoncrawl Cased
|
||||
|
||||
[UmBERTo](https://github.com/musixmatchresearch/umberto) is a Roberta-based Language Model trained on large Italian Corpora and uses two innovative approaches: SentencePiece and Whole Word Masking. Now available at [github.com/huggingface/transformers](https://huggingface.co/Musixmatch/umberto-commoncrawl-cased-v1)
|
||||
|
||||
<p align="center">
|
||||
<img src="https://user-images.githubusercontent.com/7140210/72913702-d55a8480-3d3d-11ea-99fc-f2ef29af4e72.jpg" width="700"> </br>
|
||||
Marco Lodola, Monument to Umberto Eco, Alessandria 2019
|
||||
</p>
|
||||
|
||||
## Dataset
|
||||
UmBERTo-Commoncrawl-Cased utilizes the Italian subcorpus of [OSCAR](https://traces1.inria.fr/oscar/) as training set of the language model. We used deduplicated version of the Italian corpus that consists in 70 GB of plain text data, 210M sentences with 11B words where the sentences have been filtered and shuffled at line level in order to be used for NLP research.
|
||||
|
||||
## Pre-trained model
|
||||
|
||||
| Model | WWM | Cased | Tokenizer | Vocab Size | Train Steps | Download |
|
||||
| ------ | ------ | ------ | ------ | ------ |------ | ------ |
|
||||
| `umberto-commoncrawl-cased-v1` | YES | YES | SPM | 32K | 125k | [Link](http://bit.ly/35zO7GH) |
|
||||
|
||||
This model was trained with [SentencePiece](https://github.com/google/sentencepiece) and Whole Word Masking.
|
||||
|
||||
## Downstream Tasks
|
||||
These results refers to umberto-commoncrawl-cased model. All details are at [Umberto](https://github.com/musixmatchresearch/umberto) Official Page.
|
||||
|
||||
#### Named Entity Recognition (NER)
|
||||
|
||||
| Dataset | F1 | Precision | Recall | Accuracy |
|
||||
| ------ | ------ | ------ | ------ | ------ |
|
||||
| **ICAB-EvalITA07** | **87.565** | 86.596 | 88.556 | 98.690 |
|
||||
| **WikiNER-ITA** | **92.531** | 92.509 | 92.553 | 99.136 |
|
||||
|
||||
#### Part of Speech (POS)
|
||||
|
||||
| Dataset | F1 | Precision | Recall | Accuracy |
|
||||
| ------ | ------ | ------ | ------ | ------ |
|
||||
| **UD_Italian-ISDT** | 98.870 | 98.861 | 98.879 | **98.977** |
|
||||
| **UD_Italian-ParTUT** | 98.786 | 98.812 | 98.760 | **98.903** |
|
||||
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
##### Load UmBERTo with AutoModel, Autotokenizer:
|
||||
|
||||
```python
|
||||
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModel
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("Musixmatch/umberto-commoncrawl-cased-v1")
|
||||
umberto = AutoModel.from_pretrained("Musixmatch/umberto-commoncrawl-cased-v1")
|
||||
|
||||
encoded_input = tokenizer.encode("Umberto Eco è stato un grande scrittore")
|
||||
input_ids = torch.tensor(encoded_input).unsqueeze(0) # Batch size 1
|
||||
outputs = umberto(input_ids)
|
||||
last_hidden_states = outputs[0] # The last hidden-state is the first element of the output
|
||||
```
|
||||
|
||||
##### Predict masked token:
|
||||
|
||||
```python
|
||||
from transformers import pipeline
|
||||
|
||||
fill_mask = pipeline(
|
||||
"fill-mask",
|
||||
model="Musixmatch/umberto-commoncrawl-cased-v1",
|
||||
tokenizer="Musixmatch/umberto-commoncrawl-cased-v1"
|
||||
)
|
||||
|
||||
result = fill_mask("Umberto Eco è <mask> un grande scrittore")
|
||||
# {'sequence': '<s> Umberto Eco è considerato un grande scrittore</s>', 'score': 0.18599839508533478, 'token': 5032}
|
||||
# {'sequence': '<s> Umberto Eco è stato un grande scrittore</s>', 'score': 0.17816807329654694, 'token': 471}
|
||||
# {'sequence': '<s> Umberto Eco è sicuramente un grande scrittore</s>', 'score': 0.16565583646297455, 'token': 2654}
|
||||
# {'sequence': '<s> Umberto Eco è indubbiamente un grande scrittore</s>', 'score': 0.0932890921831131, 'token': 17908}
|
||||
# {'sequence': '<s> Umberto Eco è certamente un grande scrittore</s>', 'score': 0.054701317101716995, 'token': 5269}
|
||||
```
|
||||
|
||||
|
||||
## Citation
|
||||
All of the original datasets are publicly available or were released with the owners' grant. The datasets are all released under a CC0 or CCBY license.
|
||||
|
||||
* UD Italian-ISDT Dataset [Github](https://github.com/UniversalDependencies/UD_Italian-ISDT)
|
||||
* UD Italian-ParTUT Dataset [Github](https://github.com/UniversalDependencies/UD_Italian-ParTUT)
|
||||
* I-CAB (Italian Content Annotation Bank), EvalITA [Page](http://www.evalita.it/)
|
||||
* WIKINER [Page](https://figshare.com/articles/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500) , [Paper](https://www.sciencedirect.com/science/article/pii/S0004370212000276?via%3Dihub)
|
||||
|
||||
```
|
||||
@inproceedings {magnini2006annotazione,
|
||||
title = {Annotazione di contenuti concettuali in un corpus italiano: I - CAB},
|
||||
author = {Magnini,Bernardo and Cappelli,Amedeo and Pianta,Emanuele and Speranza,Manuela and Bartalesi Lenzi,V and Sprugnoli,Rachele and Romano,Lorenza and Girardi,Christian and Negri,Matteo},
|
||||
booktitle = {Proc.of SILFI 2006},
|
||||
year = {2006}
|
||||
}
|
||||
@inproceedings {magnini2006cab,
|
||||
title = {I - CAB: the Italian Content Annotation Bank.},
|
||||
author = {Magnini,Bernardo and Pianta,Emanuele and Girardi,Christian and Negri,Matteo and Romano,Lorenza and Speranza,Manuela and Lenzi,Valentina Bartalesi and Sprugnoli,Rachele},
|
||||
booktitle = {LREC},
|
||||
pages = {963--968},
|
||||
year = {2006},
|
||||
organization = {Citeseer}
|
||||
}
|
||||
```
|
||||
|
||||
## Authors
|
||||
|
||||
**Loreto Parisi**: `loreto at musixmatch dot com`, [loretoparisi](https://github.com/loretoparisi)<br>
|
||||
**Simone Francia**: `simone.francia at musixmatch dot com`, [simonefrancia](https://github.com/simonefrancia)<br>
|
||||
**Paolo Magnani**: `paul.magnani95 at gmail dot com`, [paulthemagno](https://github.com/paulthemagno)<br>
|
||||
|
||||
## About Musixmatch AI
|
||||
<br>
|
||||
We do Machine Learning and Artificial Intelligence @[musixmatch](https://twitter.com/Musixmatch)<br>
|
||||
Follow us on [Twitter](https://twitter.com/musixmatchai) [Github](https://github.com/musixmatchresearch)
|
||||
|
||||
|
||||
@@ -1,113 +0,0 @@
|
||||
# UmBERTo Wikipedia Uncased
|
||||
|
||||
[UmBERTo](https://github.com/musixmatchresearch/umberto) is a Roberta-based Language Model trained on large Italian Corpora and uses two innovative approaches: SentencePiece and Whole Word Masking. Now available at [github.com/huggingface/transformers](https://huggingface.co/Musixmatch/umberto-commoncrawl-cased-v1)
|
||||
|
||||
<p align="center">
|
||||
<img src="https://user-images.githubusercontent.com/7140210/72913702-d55a8480-3d3d-11ea-99fc-f2ef29af4e72.jpg" width="700"> </br>
|
||||
Marco Lodola, Monument to Umberto Eco, Alessandria 2019
|
||||
</p>
|
||||
|
||||
## Dataset
|
||||
UmBERTo-Wikipedia-Uncased Training is trained on a relative small corpus (~7GB) extracted from [Wikipedia-ITA](https://linguatools.org/tools/corpora/wikipedia-monolingual-corpora/).
|
||||
|
||||
## Pre-trained model
|
||||
|
||||
| Model | WWM | Cased | Tokenizer | Vocab Size | Train Steps | Download |
|
||||
| ------ | ------ | ------ | ------ | ------ |------ | ------ |
|
||||
| `umberto-wikipedia-uncased-v1` | YES | YES | SPM | 32K | 100k | [Link](http://bit.ly/35wbSj6) |
|
||||
|
||||
This model was trained with [SentencePiece](https://github.com/google/sentencepiece) and Whole Word Masking.
|
||||
|
||||
## Downstream Tasks
|
||||
These results refers to umberto-wikipedia-uncased model. All details are at [Umberto](https://github.com/musixmatchresearch/umberto) Official Page.
|
||||
|
||||
#### Named Entity Recognition (NER)
|
||||
|
||||
| Dataset | F1 | Precision | Recall | Accuracy |
|
||||
| ------ | ------ | ------ | ------ | ----- |
|
||||
| **ICAB-EvalITA07** | **86.240** | 85.939 | 86.544 | 98.534 |
|
||||
| **WikiNER-ITA** | **90.483** | 90.328 | 90.638 | 98.661 |
|
||||
|
||||
#### Part of Speech (POS)
|
||||
|
||||
| Dataset | F1 | Precision | Recall | Accuracy |
|
||||
| ------ | ------ | ------ | ------ | ------ |
|
||||
| **UD_Italian-ISDT** | 98.563 | 98.508 | 98.618 | **98.717** |
|
||||
| **UD_Italian-ParTUT** | 97.810 | 97.835 | 97.784 | **98.060** |
|
||||
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
##### Load UmBERTo Wikipedia Uncased with AutoModel, Autotokenizer:
|
||||
|
||||
```python
|
||||
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModel
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("Musixmatch/umberto-wikipedia-uncased-v1")
|
||||
umberto = AutoModel.from_pretrained("Musixmatch/umberto-wikipedia-uncased-v1")
|
||||
|
||||
encoded_input = tokenizer.encode("Umberto Eco è stato un grande scrittore")
|
||||
input_ids = torch.tensor(encoded_input).unsqueeze(0) # Batch size 1
|
||||
outputs = umberto(input_ids)
|
||||
last_hidden_states = outputs[0] # The last hidden-state is the first element of the output
|
||||
```
|
||||
|
||||
##### Predict masked token:
|
||||
|
||||
```python
|
||||
from transformers import pipeline
|
||||
|
||||
fill_mask = pipeline(
|
||||
"fill-mask",
|
||||
model="Musixmatch/umberto-wikipedia-uncased-v1",
|
||||
tokenizer="Musixmatch/umberto-wikipedia-uncased-v1"
|
||||
)
|
||||
|
||||
result = fill_mask("Umberto Eco è <mask> un grande scrittore")
|
||||
# {'sequence': '<s> umberto eco è stato un grande scrittore</s>', 'score': 0.5784581303596497, 'token': 361}
|
||||
# {'sequence': '<s> umberto eco è anche un grande scrittore</s>', 'score': 0.33813193440437317, 'token': 269}
|
||||
# {'sequence': '<s> umberto eco è considerato un grande scrittore</s>', 'score': 0.027196012437343597, 'token': 3236}
|
||||
# {'sequence': '<s> umberto eco è diventato un grande scrittore</s>', 'score': 0.013716378249228, 'token': 5742}
|
||||
# {'sequence': '<s> umberto eco è inoltre un grande scrittore</s>', 'score': 0.010662357322871685, 'token': 1030}
|
||||
```
|
||||
|
||||
|
||||
## Citation
|
||||
All of the original datasets are publicly available or were released with the owners' grant. The datasets are all released under a CC0 or CCBY license.
|
||||
|
||||
* UD Italian-ISDT Dataset [Github](https://github.com/UniversalDependencies/UD_Italian-ISDT)
|
||||
* UD Italian-ParTUT Dataset [Github](https://github.com/UniversalDependencies/UD_Italian-ParTUT)
|
||||
* I-CAB (Italian Content Annotation Bank), EvalITA [Page](http://www.evalita.it/)
|
||||
* WIKINER [Page](https://figshare.com/articles/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500) , [Paper](https://www.sciencedirect.com/science/article/pii/S0004370212000276?via%3Dihub)
|
||||
|
||||
```
|
||||
@inproceedings {magnini2006annotazione,
|
||||
title = {Annotazione di contenuti concettuali in un corpus italiano: I - CAB},
|
||||
author = {Magnini,Bernardo and Cappelli,Amedeo and Pianta,Emanuele and Speranza,Manuela and Bartalesi Lenzi,V and Sprugnoli,Rachele and Romano,Lorenza and Girardi,Christian and Negri,Matteo},
|
||||
booktitle = {Proc.of SILFI 2006},
|
||||
year = {2006}
|
||||
}
|
||||
@inproceedings {magnini2006cab,
|
||||
title = {I - CAB: the Italian Content Annotation Bank.},
|
||||
author = {Magnini,Bernardo and Pianta,Emanuele and Girardi,Christian and Negri,Matteo and Romano,Lorenza and Speranza,Manuela and Lenzi,Valentina Bartalesi and Sprugnoli,Rachele},
|
||||
booktitle = {LREC},
|
||||
pages = {963--968},
|
||||
year = {2006},
|
||||
organization = {Citeseer}
|
||||
}
|
||||
```
|
||||
|
||||
## Authors
|
||||
|
||||
**Loreto Parisi**: `loreto at musixmatch dot com`, [loretoparisi](https://github.com/loretoparisi)<br>
|
||||
**Simone Francia**: `simone.francia at musixmatch dot com`, [simonefrancia](https://github.com/simonefrancia)<br>
|
||||
**Paolo Magnani**: `paul.magnani95 at gmail dot com`, [paulthemagno](https://github.com/paulthemagno)<br>
|
||||
|
||||
## About Musixmatch AI
|
||||
<br>
|
||||
We do Machine Learning and Artificial Intelligence @[musixmatch](https://twitter.com/Musixmatch)<br>
|
||||
Follow us on [Twitter](https://twitter.com/musixmatchai) [Github](https://github.com/musixmatchresearch)
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
# 🤗 + 📚 dbmdz German BERT models
|
||||
|
||||
In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
|
||||
Library open sources another German BERT models 🎉
|
||||
|
||||
# German BERT
|
||||
|
||||
## Stats
|
||||
|
||||
In addition to the recently released [German BERT](https://deepset.ai/german-bert)
|
||||
model by [deepset](https://deepset.ai/) we provide another German-language model.
|
||||
|
||||
The source data for the model consists of a recent Wikipedia dump, EU Bookshop corpus,
|
||||
Open Subtitles, CommonCrawl, ParaCrawl and News Crawl. This results in a dataset with
|
||||
a size of 16GB and 2,350,234,427 tokens.
|
||||
|
||||
For sentence splitting, we use [spacy](https://spacy.io/). Our preprocessing steps
|
||||
(sentence piece model for vocab generation) follow those used for training
|
||||
[SciBERT](https://github.com/allenai/scibert). The model is trained with an initial
|
||||
sequence length of 512 subwords and was performed for 1.5M steps.
|
||||
|
||||
This release includes both cased and uncased models.
|
||||
|
||||
## Model weights
|
||||
|
||||
Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
|
||||
compatible weights are available. If you need access to TensorFlow checkpoints,
|
||||
please raise an issue!
|
||||
|
||||
| Model | Downloads
|
||||
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------
|
||||
| `bert-base-german-dbmdz-cased` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-config.json) • [`pytorch_model.bin`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-pytorch_model.bin) • [`vocab.txt`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-vocab.txt)
|
||||
| `bert-base-german-dbmdz-uncased` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-config.json) • [`pytorch_model.bin`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-pytorch_model.bin) • [`vocab.txt`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-vocab.txt)
|
||||
|
||||
## Usage
|
||||
|
||||
With Transformers >= 2.3 our German BERT models can be loaded like:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased")
|
||||
model = AutoModel.from_pretrained("dbmdz/bert-base-german-cased")
|
||||
```
|
||||
|
||||
## Results
|
||||
|
||||
For results on downstream tasks like NER or PoS tagging, please refer to
|
||||
[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
|
||||
|
||||
# Huggingface model hub
|
||||
|
||||
All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
|
||||
|
||||
# Contact (Bugs, Feedback, Contribution and more)
|
||||
|
||||
For questions about our BERT models just open an issue
|
||||
[here](https://github.com/dbmdz/berts/issues/new) 🤗
|
||||
|
||||
# Acknowledgments
|
||||
|
||||
Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
|
||||
Thanks for providing access to the TFRC ❤️
|
||||
|
||||
Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
|
||||
it is possible to download both cased and uncased models from their S3 storage 🤗
|
||||
@@ -1,66 +0,0 @@
|
||||
# 🤗 + 📚 dbmdz German BERT models
|
||||
|
||||
In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
|
||||
Library open sources another German BERT models 🎉
|
||||
|
||||
# German BERT
|
||||
|
||||
## Stats
|
||||
|
||||
In addition to the recently released [German BERT](https://deepset.ai/german-bert)
|
||||
model by [deepset](https://deepset.ai/) we provide another German-language model.
|
||||
|
||||
The source data for the model consists of a recent Wikipedia dump, EU Bookshop corpus,
|
||||
Open Subtitles, CommonCrawl, ParaCrawl and News Crawl. This results in a dataset with
|
||||
a size of 16GB and 2,350,234,427 tokens.
|
||||
|
||||
For sentence splitting, we use [spacy](https://spacy.io/). Our preprocessing steps
|
||||
(sentence piece model for vocab generation) follow those used for training
|
||||
[SciBERT](https://github.com/allenai/scibert). The model is trained with an initial
|
||||
sequence length of 512 subwords and was performed for 1.5M steps.
|
||||
|
||||
This release includes both cased and uncased models.
|
||||
|
||||
## Model weights
|
||||
|
||||
Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
|
||||
compatible weights are available. If you need access to TensorFlow checkpoints,
|
||||
please raise an issue!
|
||||
|
||||
| Model | Downloads
|
||||
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------
|
||||
| `bert-base-german-dbmdz-cased` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-config.json) • [`pytorch_model.bin`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-pytorch_model.bin) • [`vocab.txt`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-vocab.txt)
|
||||
| `bert-base-german-dbmdz-uncased` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-config.json) • [`pytorch_model.bin`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-pytorch_model.bin) • [`vocab.txt`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-vocab.txt)
|
||||
|
||||
## Usage
|
||||
|
||||
With Transformers >= 2.3 our German BERT models can be loaded like:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased")
|
||||
model = AutoModel.from_pretrained("dbmdz/bert-base-german-cased")
|
||||
```
|
||||
|
||||
## Results
|
||||
|
||||
For results on downstream tasks like NER or PoS tagging, please refer to
|
||||
[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
|
||||
|
||||
# Huggingface model hub
|
||||
|
||||
All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
|
||||
|
||||
# Contact (Bugs, Feedback, Contribution and more)
|
||||
|
||||
For questions about our BERT models just open an issue
|
||||
[here](https://github.com/dbmdz/berts/issues/new) 🤗
|
||||
|
||||
# Acknowledgments
|
||||
|
||||
Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
|
||||
Thanks for providing access to the TFRC ❤️
|
||||
|
||||
Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
|
||||
it is possible to download both cased and uncased models from their S3 storage 🤗
|
||||
@@ -1,73 +0,0 @@
|
||||
# 🤗 + 📚 dbmdz BERT models
|
||||
|
||||
In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
|
||||
Library open sources Italian BERT models 🎉
|
||||
|
||||
# Italian BERT
|
||||
|
||||
The source data for the Italian BERT model consists of a recent Wikipedia dump and
|
||||
various texts from the [OPUS corpora](http://opus.nlpl.eu/) collection. The final
|
||||
training corpus has a size of 13GB and 2,050,057,573 tokens.
|
||||
|
||||
For sentence splitting, we use NLTK (faster compared to spacy).
|
||||
Our cased and uncased models are training with an initial sequence length of 512
|
||||
subwords for ~2-3M steps.
|
||||
|
||||
For the XXL Italian models, we use the same training data from OPUS and extend
|
||||
it with data from the Italian part of the [OSCAR corpus](https://traces1.inria.fr/oscar/).
|
||||
Thus, the final training corpus has a size of 81GB and 13,138,379,147 tokens.
|
||||
|
||||
## Model weights
|
||||
|
||||
Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
|
||||
compatible weights are available. If you need access to TensorFlow checkpoints,
|
||||
please raise an issue!
|
||||
|
||||
| Model | Downloads
|
||||
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------
|
||||
| `dbmdz/bert-base-italian-cased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-uncased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-xxl-cased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-xxl-uncased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/vocab.txt)
|
||||
|
||||
## Results
|
||||
|
||||
For results on downstream tasks like NER or PoS tagging, please refer to
|
||||
[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
|
||||
|
||||
## Usage
|
||||
|
||||
With Transformers >= 2.3 our Italian BERT models can be loaded like:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-cased")
|
||||
model = AutoModel.from_pretrained("dbmdz/bert-base-italian-cased")
|
||||
```
|
||||
|
||||
To load the (recommended) Italian XXL BERT models, just use:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
|
||||
model = AutoModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
|
||||
```
|
||||
|
||||
# Huggingface model hub
|
||||
|
||||
All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
|
||||
|
||||
# Contact (Bugs, Feedback, Contribution and more)
|
||||
|
||||
For questions about our BERT models just open an issue
|
||||
[here](https://github.com/dbmdz/berts/issues/new) 🤗
|
||||
|
||||
# Acknowledgments
|
||||
|
||||
Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
|
||||
Thanks for providing access to the TFRC ❤️
|
||||
|
||||
Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
|
||||
it is possible to download both cased and uncased models from their S3 storage 🤗
|
||||
@@ -1,73 +0,0 @@
|
||||
# 🤗 + 📚 dbmdz BERT models
|
||||
|
||||
In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
|
||||
Library open sources Italian BERT models 🎉
|
||||
|
||||
# Italian BERT
|
||||
|
||||
The source data for the Italian BERT model consists of a recent Wikipedia dump and
|
||||
various texts from the [OPUS corpora](http://opus.nlpl.eu/) collection. The final
|
||||
training corpus has a size of 13GB and 2,050,057,573 tokens.
|
||||
|
||||
For sentence splitting, we use NLTK (faster compared to spacy).
|
||||
Our cased and uncased models are training with an initial sequence length of 512
|
||||
subwords for ~2-3M steps.
|
||||
|
||||
For the XXL Italian models, we use the same training data from OPUS and extend
|
||||
it with data from the Italian part of the [OSCAR corpus](https://traces1.inria.fr/oscar/).
|
||||
Thus, the final training corpus has a size of 81GB and 13,138,379,147 tokens.
|
||||
|
||||
## Model weights
|
||||
|
||||
Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
|
||||
compatible weights are available. If you need access to TensorFlow checkpoints,
|
||||
please raise an issue!
|
||||
|
||||
| Model | Downloads
|
||||
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------
|
||||
| `dbmdz/bert-base-italian-cased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-uncased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-xxl-cased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-xxl-uncased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/vocab.txt)
|
||||
|
||||
## Results
|
||||
|
||||
For results on downstream tasks like NER or PoS tagging, please refer to
|
||||
[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
|
||||
|
||||
## Usage
|
||||
|
||||
With Transformers >= 2.3 our Italian BERT models can be loaded like:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-cased")
|
||||
model = AutoModel.from_pretrained("dbmdz/bert-base-italian-cased")
|
||||
```
|
||||
|
||||
To load the (recommended) Italian XXL BERT models, just use:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
|
||||
model = AutoModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
|
||||
```
|
||||
|
||||
# Huggingface model hub
|
||||
|
||||
All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
|
||||
|
||||
# Contact (Bugs, Feedback, Contribution and more)
|
||||
|
||||
For questions about our BERT models just open an issue
|
||||
[here](https://github.com/dbmdz/berts/issues/new) 🤗
|
||||
|
||||
# Acknowledgments
|
||||
|
||||
Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
|
||||
Thanks for providing access to the TFRC ❤️
|
||||
|
||||
Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
|
||||
it is possible to download both cased and uncased models from their S3 storage 🤗
|
||||
@@ -1,73 +0,0 @@
|
||||
# 🤗 + 📚 dbmdz BERT models
|
||||
|
||||
In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
|
||||
Library open sources Italian BERT models 🎉
|
||||
|
||||
# Italian BERT
|
||||
|
||||
The source data for the Italian BERT model consists of a recent Wikipedia dump and
|
||||
various texts from the [OPUS corpora](http://opus.nlpl.eu/) collection. The final
|
||||
training corpus has a size of 13GB and 2,050,057,573 tokens.
|
||||
|
||||
For sentence splitting, we use NLTK (faster compared to spacy).
|
||||
Our cased and uncased models are training with an initial sequence length of 512
|
||||
subwords for ~2-3M steps.
|
||||
|
||||
For the XXL Italian models, we use the same training data from OPUS and extend
|
||||
it with data from the Italian part of the [OSCAR corpus](https://traces1.inria.fr/oscar/).
|
||||
Thus, the final training corpus has a size of 81GB and 13,138,379,147 tokens.
|
||||
|
||||
## Model weights
|
||||
|
||||
Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
|
||||
compatible weights are available. If you need access to TensorFlow checkpoints,
|
||||
please raise an issue!
|
||||
|
||||
| Model | Downloads
|
||||
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------
|
||||
| `dbmdz/bert-base-italian-cased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-uncased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-xxl-cased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-xxl-uncased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/vocab.txt)
|
||||
|
||||
## Results
|
||||
|
||||
For results on downstream tasks like NER or PoS tagging, please refer to
|
||||
[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
|
||||
|
||||
## Usage
|
||||
|
||||
With Transformers >= 2.3 our Italian BERT models can be loaded like:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-cased")
|
||||
model = AutoModel.from_pretrained("dbmdz/bert-base-italian-cased")
|
||||
```
|
||||
|
||||
To load the (recommended) Italian XXL BERT models, just use:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
|
||||
model = AutoModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
|
||||
```
|
||||
|
||||
# Huggingface model hub
|
||||
|
||||
All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
|
||||
|
||||
# Contact (Bugs, Feedback, Contribution and more)
|
||||
|
||||
For questions about our BERT models just open an issue
|
||||
[here](https://github.com/dbmdz/berts/issues/new) 🤗
|
||||
|
||||
# Acknowledgments
|
||||
|
||||
Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
|
||||
Thanks for providing access to the TFRC ❤️
|
||||
|
||||
Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
|
||||
it is possible to download both cased and uncased models from their S3 storage 🤗
|
||||
@@ -1,73 +0,0 @@
|
||||
# 🤗 + 📚 dbmdz BERT models
|
||||
|
||||
In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
|
||||
Library open sources Italian BERT models 🎉
|
||||
|
||||
# Italian BERT
|
||||
|
||||
The source data for the Italian BERT model consists of a recent Wikipedia dump and
|
||||
various texts from the [OPUS corpora](http://opus.nlpl.eu/) collection. The final
|
||||
training corpus has a size of 13GB and 2,050,057,573 tokens.
|
||||
|
||||
For sentence splitting, we use NLTK (faster compared to spacy).
|
||||
Our cased and uncased models are training with an initial sequence length of 512
|
||||
subwords for ~2-3M steps.
|
||||
|
||||
For the XXL Italian models, we use the same training data from OPUS and extend
|
||||
it with data from the Italian part of the [OSCAR corpus](https://traces1.inria.fr/oscar/).
|
||||
Thus, the final training corpus has a size of 81GB and 13,138,379,147 tokens.
|
||||
|
||||
## Model weights
|
||||
|
||||
Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
|
||||
compatible weights are available. If you need access to TensorFlow checkpoints,
|
||||
please raise an issue!
|
||||
|
||||
| Model | Downloads
|
||||
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------
|
||||
| `dbmdz/bert-base-italian-cased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-uncased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-xxl-cased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/vocab.txt)
|
||||
| `dbmdz/bert-base-italian-xxl-uncased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/vocab.txt)
|
||||
|
||||
## Results
|
||||
|
||||
For results on downstream tasks like NER or PoS tagging, please refer to
|
||||
[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).
|
||||
|
||||
## Usage
|
||||
|
||||
With Transformers >= 2.3 our Italian BERT models can be loaded like:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-cased")
|
||||
model = AutoModel.from_pretrained("dbmdz/bert-base-italian-cased")
|
||||
```
|
||||
|
||||
To load the (recommended) Italian XXL BERT models, just use:
|
||||
|
||||
```python
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
|
||||
model = AutoModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
|
||||
```
|
||||
|
||||
# Huggingface model hub
|
||||
|
||||
All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
|
||||
|
||||
# Contact (Bugs, Feedback, Contribution and more)
|
||||
|
||||
For questions about our BERT models just open an issue
|
||||
[here](https://github.com/dbmdz/berts/issues/new) 🤗
|
||||
|
||||
# Acknowledgments
|
||||
|
||||
Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
|
||||
Thanks for providing access to the TFRC ❤️
|
||||
|
||||
Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
|
||||
it is possible to download both cased and uncased models from their S3 storage 🤗
|
||||
@@ -1,46 +0,0 @@
|
||||
# Multilingual + Dutch SQuAD2.0
|
||||
|
||||
This model is the multilingual model provided by the Google research team with a fine-tuned dutch Q&A downstream task.
|
||||
|
||||
## Details of the language model(bert-base-multilingual-cased)
|
||||
|
||||
Language model ([**bert-base-multilingual-cased**](https://github.com/google-research/bert/blob/master/multilingual.md)):
|
||||
12-layer, 768-hidden, 12-heads, 110M parameters.
|
||||
Trained on cased text in the top 104 languages with the largest Wikipedias.
|
||||
|
||||
## Details of the downstream task - Dataset
|
||||
Using the `mtranslate` Python module, [**SQuAD2.0**](https://rajpurkar.github.io/SQuAD-explorer/) was machine-translated. In order to find the start tokens the direct translations of the answers were searched in the corresponding paragraphs. Since the answer could not always be found in the text, due to the different translations depending on the context (missing context in the pure answer), a loss of question-answer examples occurred. This is a potential problem where errors can occur in the data set (but in the end it was a quick and dirty solution that worked well enough for my task).
|
||||
|
||||
| Dataset | # Q&A |
|
||||
| ---------------------- | ----- |
|
||||
| SQuAD2.0 Train | 130 K |
|
||||
| Dutch SQuAD2.0 Train | 99 K |
|
||||
| SQuAD2.0 Dev | 12 K |
|
||||
| Dutch SQuAD2.0 Dev | 10 K |
|
||||
|
||||
## Model training
|
||||
|
||||
The model was trained on a Tesla V100 GPU with the following command:
|
||||
|
||||
```python
|
||||
export SQUAD_DIR=path/to/nl_squad
|
||||
|
||||
python run_squad.py \
|
||||
--model_type bert \
|
||||
--model_name_or_path bert-base-multilingual-cased \
|
||||
--version_2_with_negative \
|
||||
--do_train \
|
||||
--do_eval \
|
||||
--train_file $SQUAD_DIR/train_nl-v2.0.json \
|
||||
--predict_file $SQUAD_DIR/dev_nl-v2.0.json \
|
||||
--per_gpu_train_batch_size 12 \
|
||||
--learning_rate 3e-5 \
|
||||
--num_train_epochs 2.0 \
|
||||
--max_seq_length 384 \
|
||||
--doc_stride 128 \
|
||||
--output_dir /tmp/output_dir/
|
||||
```
|
||||
|
||||
**Results**:
|
||||
|
||||
{'exact': **67.38**, 'f1': **71.36**}
|
||||
@@ -1,31 +0,0 @@
|
||||
# Tensorflow CamemBERT
|
||||
|
||||
In this repository you will find different versions of the CamemBERT model for Tensorflow.
|
||||
|
||||
## CamemBERT
|
||||
|
||||
[CamemBERT](https://camembert-model.fr/) is a state-of-the-art language model for French based on the RoBERTa architecture pretrained on the French subcorpus of the newly available multilingual corpus OSCAR.
|
||||
|
||||
## Model Weights
|
||||
|
||||
| Model | Downloads
|
||||
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------
|
||||
| `jplu/tf-camembert-base` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-camembert-base/config.json) • [`tf_model.h5`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-camembert-base/tf_model.h5)
|
||||
|
||||
## Usage
|
||||
|
||||
With Transformers >= 2.4 the Tensorflow models of CamemBERT can be loaded like:
|
||||
|
||||
```python
|
||||
from transformers import TFCamembertModel
|
||||
|
||||
model = TFCamembertModel.from_pretrained("jplu/tf-camembert-base")
|
||||
```
|
||||
|
||||
## Huggingface model hub
|
||||
|
||||
All models are available on the [Huggingface model hub](https://huggingface.co/jplu).
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Thanks to all the Huggingface team for the support and their amazing library!
|
||||
@@ -1,36 +0,0 @@
|
||||
# Tensorflow XLM-RoBERTa
|
||||
|
||||
In this repository you will find different versions of the XLM-RoBERTa model for Tensorflow.
|
||||
|
||||
## XLM-RoBERTa
|
||||
|
||||
[XLM-RoBERTa](https://ai.facebook.com/blog/-xlm-r-state-of-the-art-cross-lingual-understanding-through-self-supervision/) is a scaled cross lingual sentence encoder. It is trained on 2.5T of data across 100 languages data filtered from Common Crawl. XLM-R achieves state-of-the-arts results on multiple cross lingual benchmarks.
|
||||
|
||||
## Model Weights
|
||||
|
||||
| Model | Downloads
|
||||
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------
|
||||
| `jplu/tf-xlm-roberta-base` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-base/config.json) • [`tf_model.h5`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-base/tf_model.h5)
|
||||
| `jplu/tf-xlm-roberta-large` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-large/config.json) • [`tf_model.h5`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-large/tf_model.h5)
|
||||
|
||||
## Usage
|
||||
|
||||
With Transformers >= 2.4 the Tensorflow models of XLM-RoBERTa can be loaded like:
|
||||
|
||||
```python
|
||||
from transformers import TFXLMRobertaModel
|
||||
|
||||
model = TFXLMRobertaModel.from_pretrained("jplu/tf-xlm-roberta-base")
|
||||
```
|
||||
Or
|
||||
```
|
||||
model = TFXLMRobertaModel.from_pretrained("jplu/tf-xlm-roberta-large")
|
||||
```
|
||||
|
||||
## Huggingface model hub
|
||||
|
||||
All models are available on the [Huggingface model hub](https://huggingface.co/jplu).
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Thanks to all the Huggingface team for the support and their amazing library!
|
||||
@@ -1,36 +0,0 @@
|
||||
# Tensorflow XLM-RoBERTa
|
||||
|
||||
In this repository you will find different versions of the XLM-RoBERTa model for Tensorflow.
|
||||
|
||||
## XLM-RoBERTa
|
||||
|
||||
[XLM-RoBERTa](https://ai.facebook.com/blog/-xlm-r-state-of-the-art-cross-lingual-understanding-through-self-supervision/) is a scaled cross lingual sentence encoder. It is trained on 2.5T of data across 100 languages data filtered from Common Crawl. XLM-R achieves state-of-the-arts results on multiple cross lingual benchmarks.
|
||||
|
||||
## Model Weights
|
||||
|
||||
| Model | Downloads
|
||||
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------
|
||||
| `jplu/tf-xlm-roberta-base` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-base/config.json) • [`tf_model.h5`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-base/tf_model.h5)
|
||||
| `jplu/tf-xlm-roberta-large` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-large/config.json) • [`tf_model.h5`](https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-large/tf_model.h5)
|
||||
|
||||
## Usage
|
||||
|
||||
With Transformers >= 2.4 the Tensorflow models of XLM-RoBERTa can be loaded like:
|
||||
|
||||
```python
|
||||
from transformers import TFXLMRobertaModel
|
||||
|
||||
model = TFXLMRobertaModel.from_pretrained("jplu/tf-xlm-roberta-base")
|
||||
```
|
||||
Or
|
||||
```
|
||||
model = TFXLMRobertaModel.from_pretrained("jplu/tf-xlm-roberta-large")
|
||||
```
|
||||
|
||||
## Huggingface model hub
|
||||
|
||||
All models are available on the [Huggingface model hub](https://huggingface.co/jplu).
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Thanks to all the Huggingface team for the support and their amazing library!
|
||||
@@ -1,25 +0,0 @@
|
||||
## How to build a dummy model
|
||||
|
||||
|
||||
```python
|
||||
from transformers.configuration_bert import BertConfig
|
||||
from transformers.modeling_bert import BertForMaskedLM
|
||||
from transformers.modeling_tf_bert import TFBertForMaskedLM
|
||||
from transformers.tokenization_bert import BertTokenizer
|
||||
|
||||
|
||||
SMALL_MODEL_IDENTIFIER = "julien-c/bert-xsmall-dummy"
|
||||
DIRNAME = "./bert-xsmall-dummy"
|
||||
|
||||
config = BertConfig(10, 20, 1, 1, 40)
|
||||
|
||||
model = BertForMaskedLM(config)
|
||||
model.save_pretrained(DIRNAME)
|
||||
|
||||
tf_model = TFBertForMaskedLM.from_pretrained(DIRNAME, from_pt=True)
|
||||
tf_model.save_pretrained(DIRNAME)
|
||||
|
||||
# Slightly different for tokenizer.
|
||||
# tokenizer = BertTokenizer.from_pretrained(DIRNAME)
|
||||
# tokenizer.save_pretrained()
|
||||
```
|
||||
@@ -1,52 +0,0 @@
|
||||
|
||||
```python
|
||||
import json
|
||||
import os
|
||||
from transformers.configuration_roberta import RobertaConfig
|
||||
from transformers import RobertaForMaskedLM, TFRobertaForMaskedLM
|
||||
|
||||
DIRNAME = "./dummy-unknown"
|
||||
|
||||
|
||||
config = RobertaConfig(10, 20, 1, 1, 40)
|
||||
|
||||
model = RobertaForMaskedLM(config)
|
||||
model.save_pretrained(DIRNAME)
|
||||
|
||||
tf_model = TFRobertaForMaskedLM.from_pretrained(DIRNAME, from_pt=True)
|
||||
tf_model.save_pretrained(DIRNAME)
|
||||
|
||||
# Tokenizer:
|
||||
|
||||
vocab = [
|
||||
"l",
|
||||
"o",
|
||||
"w",
|
||||
"e",
|
||||
"r",
|
||||
"s",
|
||||
"t",
|
||||
"i",
|
||||
"d",
|
||||
"n",
|
||||
"\u0120",
|
||||
"\u0120l",
|
||||
"\u0120n",
|
||||
"\u0120lo",
|
||||
"\u0120low",
|
||||
"er",
|
||||
"\u0120lowest",
|
||||
"\u0120newer",
|
||||
"\u0120wider",
|
||||
"<unk>",
|
||||
]
|
||||
vocab_tokens = dict(zip(vocab, range(len(vocab))))
|
||||
merges = ["#version: 0.2", "\u0120 l", "\u0120l o", "\u0120lo w", "e r", ""]
|
||||
|
||||
vocab_file = os.path.join(DIRNAME, "vocab.json")
|
||||
merges_file = os.path.join(DIRNAME, "merges.txt")
|
||||
with open(vocab_file, "w", encoding="utf-8") as fp:
|
||||
fp.write(json.dumps(vocab_tokens) + "\n")
|
||||
with open(merges_file, "w", encoding="utf-8") as fp:
|
||||
fp.write("\n".join(merges))
|
||||
```
|
||||
@@ -23,8 +23,6 @@ To create the package for pypi.
|
||||
|
||||
twine upload dist/* -r pypitest
|
||||
(pypi suggest using twine as other methods upload files via plaintext.)
|
||||
You may have to specify the repository url, use the following command then:
|
||||
twine upload dist/* -r pypitest --repository-url=https://test.pypi.org/legacy/
|
||||
|
||||
Check that you can install it in a virtualenv by running:
|
||||
pip install -i https://testpypi.python.org/pypi transformers
|
||||
@@ -65,7 +63,7 @@ extras["sklearn"] = ["scikit-learn"]
|
||||
extras["tf"] = ["tensorflow"]
|
||||
extras["torch"] = ["torch"]
|
||||
|
||||
extras["serving"] = ["pydantic", "uvicorn", "fastapi", "starlette"]
|
||||
extras["serving"] = ["pydantic", "uvicorn", "fastapi"]
|
||||
extras["all"] = extras["serving"] + ["tensorflow", "torch"]
|
||||
|
||||
extras["testing"] = ["pytest", "pytest-xdist"]
|
||||
@@ -75,7 +73,7 @@ extras["dev"] = extras["testing"] + extras["quality"] + ["mecab-python3", "sciki
|
||||
|
||||
setup(
|
||||
name="transformers",
|
||||
version="2.4.1",
|
||||
version="2.3.0",
|
||||
author="Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Google AI Language Team Authors, Open AI team Authors, Facebook AI Authors, Carnegie Mellon University Authors",
|
||||
author_email="thomas@huggingface.co",
|
||||
description="State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch",
|
||||
@@ -88,7 +86,7 @@ setup(
|
||||
packages=find_packages("src"),
|
||||
install_requires=[
|
||||
"numpy",
|
||||
"tokenizers == 0.0.11",
|
||||
"tokenizers == 0.2.1",
|
||||
# accessing files from S3 directly
|
||||
"boto3",
|
||||
# filesystem locks e.g. to prevent parallel downloads
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
# There's no way to ignore "F401 '...' imported but unused" warnings in this
|
||||
# module, but to preserve other warnings. So, don't check this module at all.
|
||||
|
||||
__version__ = "2.4.1"
|
||||
__version__ = "2.3.0"
|
||||
|
||||
# Work around to update TensorFlow's absl.logging threshold which alters the
|
||||
# default Python logging output behavior when present.
|
||||
@@ -25,7 +25,6 @@ from .configuration_bert import BERT_PRETRAINED_CONFIG_ARCHIVE_MAP, BertConfig
|
||||
from .configuration_camembert import CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, CamembertConfig
|
||||
from .configuration_ctrl import CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP, CTRLConfig
|
||||
from .configuration_distilbert import DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, DistilBertConfig
|
||||
from .configuration_flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig
|
||||
from .configuration_gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config
|
||||
from .configuration_mmbt import MMBTConfig
|
||||
from .configuration_openai import OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP, OpenAIGPTConfig
|
||||
@@ -93,7 +92,6 @@ from .modeling_tf_pytorch_utils import (
|
||||
from .pipelines import (
|
||||
CsvPipelineDataFormat,
|
||||
FeatureExtractionPipeline,
|
||||
FillMaskPipeline,
|
||||
JsonPipelineDataFormat,
|
||||
NerPipeline,
|
||||
PipedPipelineDataFormat,
|
||||
@@ -110,7 +108,6 @@ from .tokenization_bert_japanese import BertJapaneseTokenizer, CharacterTokenize
|
||||
from .tokenization_camembert import CamembertTokenizer
|
||||
from .tokenization_ctrl import CTRLTokenizer
|
||||
from .tokenization_distilbert import DistilBertTokenizer
|
||||
from .tokenization_flaubert import FlaubertTokenizer
|
||||
from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast
|
||||
from .tokenization_openai import OpenAIGPTTokenizer
|
||||
from .tokenization_roberta import RobertaTokenizer
|
||||
@@ -136,7 +133,6 @@ if is_torch_available():
|
||||
from .modeling_utils import PreTrainedModel, prune_layer, Conv1D
|
||||
from .modeling_auto import (
|
||||
AutoModel,
|
||||
AutoModelForPreTraining,
|
||||
AutoModelForSequenceClassification,
|
||||
AutoModelForQuestionAnswering,
|
||||
AutoModelWithLMHead,
|
||||
@@ -212,13 +208,6 @@ if is_torch_available():
|
||||
RobertaForQuestionAnswering,
|
||||
ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
)
|
||||
from .modeling_camembert import (
|
||||
CamembertForMaskedLM,
|
||||
CamembertModel,
|
||||
CamembertForSequenceClassification,
|
||||
CamembertForTokenClassification,
|
||||
CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
)
|
||||
from .modeling_distilbert import (
|
||||
DistilBertPreTrainedModel,
|
||||
DistilBertForMaskedLM,
|
||||
@@ -259,19 +248,9 @@ if is_torch_available():
|
||||
XLMRobertaForMultipleChoice,
|
||||
XLMRobertaForSequenceClassification,
|
||||
XLMRobertaForTokenClassification,
|
||||
XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
)
|
||||
from .modeling_mmbt import ModalEmbeddings, MMBTModel, MMBTForClassification
|
||||
|
||||
from .modeling_flaubert import (
|
||||
FlaubertModel,
|
||||
FlaubertWithLMHeadModel,
|
||||
FlaubertForSequenceClassification,
|
||||
FlaubertForQuestionAnswering,
|
||||
FlaubertForQuestionAnsweringSimple,
|
||||
FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
)
|
||||
|
||||
# Optimization
|
||||
from .optimization import (
|
||||
AdamW,
|
||||
@@ -288,7 +267,6 @@ if is_tf_available():
|
||||
from .modeling_tf_utils import TFPreTrainedModel, TFSharedEmbeddings, TFSequenceSummary, shape_list
|
||||
from .modeling_tf_auto import (
|
||||
TFAutoModel,
|
||||
TFAutoModelForPreTraining,
|
||||
TFAutoModelForSequenceClassification,
|
||||
TFAutoModelForQuestionAnswering,
|
||||
TFAutoModelWithLMHead,
|
||||
@@ -358,14 +336,6 @@ if is_tf_available():
|
||||
TF_XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
)
|
||||
|
||||
from .modeling_tf_xlm_roberta import (
|
||||
TFXLMRobertaForMaskedLM,
|
||||
TFXLMRobertaModel,
|
||||
TFXLMRobertaForSequenceClassification,
|
||||
TFXLMRobertaForTokenClassification,
|
||||
TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
)
|
||||
|
||||
from .modeling_tf_roberta import (
|
||||
TFRobertaPreTrainedModel,
|
||||
TFRobertaMainLayer,
|
||||
@@ -376,14 +346,6 @@ if is_tf_available():
|
||||
TF_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
)
|
||||
|
||||
from .modeling_tf_camembert import (
|
||||
TFCamembertModel,
|
||||
TFCamembertForMaskedLM,
|
||||
TFCamembertForSequenceClassification,
|
||||
TFCamembertForTokenClassification,
|
||||
TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
)
|
||||
|
||||
from .modeling_tf_distilbert import (
|
||||
TFDistilBertPreTrainedModel,
|
||||
TFDistilBertMainLayer,
|
||||
@@ -410,12 +372,7 @@ if is_tf_available():
|
||||
TF_ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
)
|
||||
|
||||
from .modeling_tf_t5 import (
|
||||
TFT5PreTrainedModel,
|
||||
TFT5Model,
|
||||
TFT5WithLMHeadModel,
|
||||
TF_T5_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
)
|
||||
from .modeling_tf_t5 import TFT5PreTrainedModel, TFT5Model, TFT5WithLMHeadModel, TF_T5_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
# Optimization
|
||||
from .optimization_tf import WarmUp, create_optimizer, AdamWeightDecay, GradientAccumulator
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
import platform
|
||||
from argparse import ArgumentParser
|
||||
|
||||
from transformers import __version__ as version
|
||||
from transformers import is_tf_available, is_torch_available
|
||||
from transformers.commands import BaseTransformersCLICommand
|
||||
|
||||
|
||||
def info_command_factory(_):
|
||||
return EnvironmentCommand()
|
||||
|
||||
|
||||
class EnvironmentCommand(BaseTransformersCLICommand):
|
||||
@staticmethod
|
||||
def register_subcommand(parser: ArgumentParser):
|
||||
download_parser = parser.add_parser("env")
|
||||
download_parser.set_defaults(func=info_command_factory)
|
||||
|
||||
def run(self):
|
||||
pt_version = "not installed"
|
||||
pt_cuda_available = "NA"
|
||||
if is_torch_available():
|
||||
import torch
|
||||
|
||||
pt_version = torch.__version__
|
||||
pt_cuda_available = torch.cuda.is_available()
|
||||
|
||||
tf_version = "not installed"
|
||||
tf_cuda_available = "NA"
|
||||
if is_tf_available():
|
||||
import tensorflow as tf
|
||||
|
||||
tf_version = tf.__version__
|
||||
try:
|
||||
# deprecated in v2.1
|
||||
tf_cuda_available = tf.test.is_gpu_available()
|
||||
except AttributeError:
|
||||
# returns list of devices, convert to bool
|
||||
tf_cuda_available = bool(tf.config.list_physical_devices("GPU"))
|
||||
|
||||
info = {
|
||||
"`transformers` version": version,
|
||||
"Platform": platform.platform(),
|
||||
"Python version": platform.python_version(),
|
||||
"PyTorch version (GPU?)": "{} ({})".format(pt_version, pt_cuda_available),
|
||||
"Tensorflow version (GPU?)": "{} ({})".format(tf_version, tf_cuda_available),
|
||||
"Using GPU in script?": "<fill in>",
|
||||
"Using distributed or parallel set-up in script?": "<fill in>",
|
||||
}
|
||||
|
||||
print("\nCopy-and-paste the text below in your GitHub issue and FILL OUT the two last points.\n")
|
||||
print(self.format_dict(info))
|
||||
|
||||
return info
|
||||
|
||||
@staticmethod
|
||||
def format_dict(d):
|
||||
return "\n".join(["- {}: {}".format(prop, val) for prop, val in d.items()]) + "\n"
|
||||
@@ -2,6 +2,7 @@ import logging
|
||||
from argparse import ArgumentParser, Namespace
|
||||
from typing import Any, List, Optional
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
from transformers import Pipeline
|
||||
from transformers.commands import BaseTransformersCLICommand
|
||||
from transformers.pipelines import SUPPORTED_TASKS, pipeline
|
||||
@@ -12,16 +13,15 @@ try:
|
||||
from fastapi import FastAPI, HTTPException, Body
|
||||
from fastapi.routing import APIRoute
|
||||
from pydantic import BaseModel
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
_serve_dependencies_installed = True
|
||||
_serve_dependancies_installed = True
|
||||
except (ImportError, AttributeError):
|
||||
BaseModel = object
|
||||
|
||||
def Body(*x, **y):
|
||||
pass
|
||||
|
||||
_serve_dependencies_installed = False
|
||||
_serve_dependancies_installed = False
|
||||
|
||||
|
||||
logger = logging.getLogger("transformers-cli/serving")
|
||||
@@ -111,7 +111,7 @@ class ServeCommand(BaseTransformersCLICommand):
|
||||
self.port = port
|
||||
self.workers = workers
|
||||
|
||||
if not _serve_dependencies_installed:
|
||||
if not _serve_dependancies_installed:
|
||||
raise RuntimeError(
|
||||
"Using serve command requires FastAPI and unicorn. "
|
||||
'Please install transformers with [serving]: pip install "transformers[serving]".'
|
||||
|
||||
@@ -76,8 +76,6 @@ class AlbertConfig(PretrainedConfig):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):
|
||||
The epsilon used by the layer normalization layers.
|
||||
classifier_dropout_prob (:obj:`float`, optional, defaults to 0.1):
|
||||
The dropout ratio for attached classifiers.
|
||||
|
||||
Example::
|
||||
|
||||
@@ -123,7 +121,6 @@ class AlbertConfig(PretrainedConfig):
|
||||
type_vocab_size=2,
|
||||
initializer_range=0.02,
|
||||
layer_norm_eps=1e-12,
|
||||
classifier_dropout_prob=0.1,
|
||||
**kwargs
|
||||
):
|
||||
super().__init__(**kwargs)
|
||||
@@ -143,4 +140,3 @@ class AlbertConfig(PretrainedConfig):
|
||||
self.type_vocab_size = type_vocab_size
|
||||
self.initializer_range = initializer_range
|
||||
self.layer_norm_eps = layer_norm_eps
|
||||
self.classifier_dropout_prob = classifier_dropout_prob
|
||||
|
||||
@@ -23,7 +23,6 @@ from .configuration_bert import BERT_PRETRAINED_CONFIG_ARCHIVE_MAP, BertConfig
|
||||
from .configuration_camembert import CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, CamembertConfig
|
||||
from .configuration_ctrl import CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP, CTRLConfig
|
||||
from .configuration_distilbert import DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, DistilBertConfig
|
||||
from .configuration_flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig
|
||||
from .configuration_gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config
|
||||
from .configuration_openai import OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP, OpenAIGPTConfig
|
||||
from .configuration_roberta import ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, RobertaConfig
|
||||
@@ -54,7 +53,6 @@ ALL_PRETRAINED_CONFIG_ARCHIVE_MAP = dict(
|
||||
CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
T5_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
]
|
||||
for key, value, in pretrained_map.items()
|
||||
)
|
||||
@@ -68,7 +66,6 @@ CONFIG_MAPPING = OrderedDict(
|
||||
("camembert", CamembertConfig,),
|
||||
("xlm-roberta", XLMRobertaConfig,),
|
||||
("roberta", RobertaConfig,),
|
||||
("flaubert", FlaubertConfig,),
|
||||
("bert", BertConfig,),
|
||||
("openai-gpt", OpenAIGPTConfig,),
|
||||
("gpt2", GPT2Config,),
|
||||
@@ -80,7 +77,7 @@ CONFIG_MAPPING = OrderedDict(
|
||||
)
|
||||
|
||||
|
||||
class AutoConfig:
|
||||
class AutoConfig(object):
|
||||
r"""
|
||||
:class:`~transformers.AutoConfig` is a generic configuration class
|
||||
that will be instantiated as one of the configuration classes of the library
|
||||
@@ -129,7 +126,6 @@ class AutoConfig:
|
||||
- contains `xlnet`: :class:`~transformers.XLNetConfig` (XLNet model)
|
||||
- contains `xlm`: :class:`~transformers.XLMConfig` (XLM model)
|
||||
- contains `ctrl` : :class:`~transformers.CTRLConfig` (CTRL model)
|
||||
- contains `flaubert` : :class:`~transformers.FlaubertConfig` (Flaubert model)
|
||||
|
||||
|
||||
Args:
|
||||
|
||||
@@ -45,7 +45,6 @@ BERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
||||
"bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-config.json",
|
||||
"bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/config.json",
|
||||
"bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/config.json",
|
||||
"bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/config.json",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -25,8 +25,6 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
||||
"camembert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/camembert-base-config.json",
|
||||
"umberto-commoncrawl-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/Musixmatch/umberto-commoncrawl-cased-v1/config.json",
|
||||
"umberto-wikipedia-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/Musixmatch/umberto-wikipedia-uncased-v1/config.json",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -1,152 +0,0 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2019-present CNRS, Facebook Inc. and the HuggingFace Inc. team.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
""" Flaubert configuration, based on XLM. """
|
||||
|
||||
|
||||
import logging
|
||||
|
||||
from .configuration_xlm import XLMConfig
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
||||
"flaubert-small-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_small_cased/config.json",
|
||||
"flaubert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/config.json",
|
||||
"flaubert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_cased/config.json",
|
||||
"flaubert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_large_cased/config.json",
|
||||
}
|
||||
|
||||
|
||||
class FlaubertConfig(XLMConfig):
|
||||
"""
|
||||
Configuration class to store the configuration of a `FlaubertModel`.
|
||||
This is the configuration class to store the configuration of a :class:`~transformers.XLMModel`.
|
||||
It is used to instantiate an XLM model according to the specified arguments, defining the model
|
||||
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
|
||||
the `xlm-mlm-en-2048 <https://huggingface.co/xlm-mlm-en-2048>`__ architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
|
||||
Args:
|
||||
pre_norm (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether to apply the layer normalization before or after the feed forward layer following the
|
||||
attention in each layer (Vaswani et al., Tensor2Tensor for Neural Machine Translation. 2018)
|
||||
layerdrop (:obj:`float`, `optional`, defaults to 0.0):
|
||||
Probability to drop layers during training (Fan et al., Reducing Transformer Depth on Demand
|
||||
with Structured Dropout. ICLR 2020)
|
||||
vocab_size (:obj:`int`, optional, defaults to 30145):
|
||||
Vocabulary size of the Flaubert model. Defines the different tokens that
|
||||
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.FlaubertModel`.
|
||||
emb_dim (:obj:`int`, optional, defaults to 2048):
|
||||
Dimensionality of the encoder layers and the pooler layer.
|
||||
n_layer (:obj:`int`, optional, defaults to 12):
|
||||
Number of hidden layers in the Transformer encoder.
|
||||
n_head (:obj:`int`, optional, defaults to 16):
|
||||
Number of attention heads for each attention layer in the Transformer encoder.
|
||||
dropout (:obj:`float`, optional, defaults to 0.1):
|
||||
The dropout probability for all fully connected
|
||||
layers in the embeddings, encoder, and pooler.
|
||||
attention_dropout (:obj:`float`, optional, defaults to 0.1):
|
||||
The dropout probability for the attention mechanism
|
||||
gelu_activation (:obj:`boolean`, optional, defaults to :obj:`True`):
|
||||
The non-linear activation function (function or string) in the
|
||||
encoder and pooler. If set to `True`, "gelu" will be used instead of "relu".
|
||||
sinusoidal_embeddings (:obj:`boolean`, optional, defaults to :obj:`False`):
|
||||
Whether to use sinusoidal positional embeddings instead of absolute positional embeddings.
|
||||
causal (:obj:`boolean`, optional, defaults to :obj:`False`):
|
||||
Set this to `True` for the model to behave in a causal manner.
|
||||
Causal models use a triangular attention mask in order to only attend to the left-side context instead
|
||||
if a bidirectional context.
|
||||
asm (:obj:`boolean`, optional, defaults to :obj:`False`):
|
||||
Whether to use an adaptive log softmax projection layer instead of a linear layer for the prediction
|
||||
layer.
|
||||
n_langs (:obj:`int`, optional, defaults to 1):
|
||||
The number of languages the model handles. Set to 1 for monolingual models.
|
||||
use_lang_emb (:obj:`boolean`, optional, defaults to :obj:`True`)
|
||||
Whether to use language embeddings. Some models use additional language embeddings, see
|
||||
`the multilingual models page <http://huggingface.co/transformers/multilingual.html#xlm-language-embeddings>`__
|
||||
for information on how to use them.
|
||||
max_position_embeddings (:obj:`int`, optional, defaults to 512):
|
||||
The maximum sequence length that this model might
|
||||
ever be used with. Typically set this to something large just in case
|
||||
(e.g., 512 or 1024 or 2048).
|
||||
embed_init_std (:obj:`float`, optional, defaults to 2048^-0.5):
|
||||
The standard deviation of the truncated_normal_initializer for
|
||||
initializing the embedding matrices.
|
||||
init_std (:obj:`int`, optional, defaults to 50257):
|
||||
The standard deviation of the truncated_normal_initializer for
|
||||
initializing all weight matrices except the embedding matrices.
|
||||
layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):
|
||||
The epsilon used by the layer normalization layers.
|
||||
bos_index (:obj:`int`, optional, defaults to 0):
|
||||
The index of the beginning of sentence token in the vocabulary.
|
||||
eos_index (:obj:`int`, optional, defaults to 1):
|
||||
The index of the end of sentence token in the vocabulary.
|
||||
pad_index (:obj:`int`, optional, defaults to 2):
|
||||
The index of the padding token in the vocabulary.
|
||||
unk_index (:obj:`int`, optional, defaults to 3):
|
||||
The index of the unknown token in the vocabulary.
|
||||
mask_index (:obj:`int`, optional, defaults to 5):
|
||||
The index of the masking token in the vocabulary.
|
||||
is_encoder(:obj:`boolean`, optional, defaults to :obj:`True`):
|
||||
Whether the initialized model should be a transformer encoder or decoder as seen in Vaswani et al.
|
||||
summary_type (:obj:`string`, optional, defaults to "first"):
|
||||
Argument used when doing sequence summary. Used in for the multiple choice head in
|
||||
:class:`~transformers.XLMForSequenceClassification`.
|
||||
Is one of the following options:
|
||||
- 'last' => take the last token hidden state (like XLNet)
|
||||
- 'first' => take the first token hidden state (like Bert)
|
||||
- 'mean' => take the mean of all tokens hidden states
|
||||
- 'cls_index' => supply a Tensor of classification token position (GPT/GPT-2)
|
||||
- 'attn' => Not implemented now, use multi-head attention
|
||||
summary_use_proj (:obj:`boolean`, optional, defaults to :obj:`True`):
|
||||
Argument used when doing sequence summary. Used in for the multiple choice head in
|
||||
:class:`~transformers.XLMForSequenceClassification`.
|
||||
Add a projection after the vector extraction
|
||||
summary_activation (:obj:`string` or :obj:`None`, optional, defaults to :obj:`None`):
|
||||
Argument used when doing sequence summary. Used in for the multiple choice head in
|
||||
:class:`~transformers.XLMForSequenceClassification`.
|
||||
'tanh' => add a tanh activation to the output, Other => no activation.
|
||||
summary_proj_to_labels (:obj:`boolean`, optional, defaults to :obj:`True`):
|
||||
Argument used when doing sequence summary. Used in for the multiple choice head in
|
||||
:class:`~transformers.XLMForSequenceClassification`.
|
||||
If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False.
|
||||
summary_first_dropout (:obj:`float`, optional, defaults to 0.1):
|
||||
Argument used when doing sequence summary. Used in for the multiple choice head in
|
||||
:class:`~transformers.XLMForSequenceClassification`.
|
||||
Add a dropout before the projection and activation
|
||||
start_n_top (:obj:`int`, optional, defaults to 5):
|
||||
Used in the SQuAD evaluation script for XLM and XLNet.
|
||||
end_n_top (:obj:`int`, optional, defaults to 5):
|
||||
Used in the SQuAD evaluation script for XLM and XLNet.
|
||||
mask_token_id (:obj:`int`, optional, defaults to 0):
|
||||
Model agnostic parameter to identify masked tokens when generating text in an MLM context.
|
||||
lang_id (:obj:`int`, optional, defaults to 1):
|
||||
The ID of the language used by the model. This parameter is used when generating
|
||||
text in a given language.
|
||||
"""
|
||||
|
||||
pretrained_config_archive_map = FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP
|
||||
model_type = "flaubert"
|
||||
|
||||
def __init__(self, layerdrop=0.0, pre_norm=False, **kwargs):
|
||||
"""Constructs FlaubertConfig.
|
||||
"""
|
||||
super().__init__(**kwargs)
|
||||
self.layerdrop = layerdrop
|
||||
self.pre_norm = pre_norm
|
||||
@@ -82,7 +82,6 @@ class PretrainedConfig(object):
|
||||
self.num_return_sequences = kwargs.pop("num_return_sequences", 1)
|
||||
|
||||
# Fine-tuning task arguments
|
||||
self.architectures = kwargs.pop("architectures", None)
|
||||
self.finetuning_task = kwargs.pop("finetuning_task", None)
|
||||
self.num_labels = kwargs.pop("num_labels", 2)
|
||||
self.id2label = kwargs.pop("id2label", {i: "LABEL_{}".format(i) for i in range(self.num_labels)})
|
||||
|
||||
@@ -22,7 +22,6 @@ import os
|
||||
from transformers import (
|
||||
ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
BERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
@@ -31,11 +30,9 @@ from transformers import (
|
||||
T5_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
XLM_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
AlbertConfig,
|
||||
BertConfig,
|
||||
CamembertConfig,
|
||||
CTRLConfig,
|
||||
DistilBertConfig,
|
||||
GPT2Config,
|
||||
@@ -46,7 +43,6 @@ from transformers import (
|
||||
TFBertForPreTraining,
|
||||
TFBertForQuestionAnswering,
|
||||
TFBertForSequenceClassification,
|
||||
TFCamembertForMaskedLM,
|
||||
TFCTRLLMHeadModel,
|
||||
TFDistilBertForMaskedLM,
|
||||
TFDistilBertForQuestionAnswering,
|
||||
@@ -56,12 +52,10 @@ from transformers import (
|
||||
TFRobertaForSequenceClassification,
|
||||
TFT5WithLMHeadModel,
|
||||
TFTransfoXLLMHeadModel,
|
||||
TFXLMRobertaForMaskedLM,
|
||||
TFXLMWithLMHeadModel,
|
||||
TFXLNetLMHeadModel,
|
||||
TransfoXLConfig,
|
||||
XLMConfig,
|
||||
XLMRobertaConfig,
|
||||
XLNetConfig,
|
||||
cached_path,
|
||||
is_torch_available,
|
||||
@@ -83,8 +77,6 @@ if is_torch_available():
|
||||
XLNET_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
XLMWithLMHeadModel,
|
||||
XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
XLMRobertaForMaskedLM,
|
||||
TransfoXLLMHeadModel,
|
||||
TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
OpenAIGPTLMHeadModel,
|
||||
@@ -92,9 +84,6 @@ if is_torch_available():
|
||||
RobertaForMaskedLM,
|
||||
RobertaForSequenceClassification,
|
||||
ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
CamembertForMaskedLM,
|
||||
CamembertForSequenceClassification,
|
||||
CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
DistilBertForMaskedLM,
|
||||
DistilBertForQuestionAnswering,
|
||||
DistilBertForSequenceClassification,
|
||||
@@ -118,8 +107,6 @@ else:
|
||||
XLNET_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
XLMWithLMHeadModel,
|
||||
XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
XLMRobertaForMaskedLM,
|
||||
TransfoXLLMHeadModel,
|
||||
TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
OpenAIGPTLMHeadModel,
|
||||
@@ -127,9 +114,6 @@ else:
|
||||
RobertaForMaskedLM,
|
||||
RobertaForSequenceClassification,
|
||||
ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
CamembertForMaskedLM,
|
||||
CamembertForSequenceClassification,
|
||||
CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
DistilBertForMaskedLM,
|
||||
DistilBertForSequenceClassification,
|
||||
DistilBertForQuestionAnswering,
|
||||
@@ -168,11 +152,6 @@ else:
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
)
|
||||
|
||||
|
||||
@@ -228,13 +207,6 @@ MODEL_CLASSES = {
|
||||
XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
XLM_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
),
|
||||
"xlm-roberta": (
|
||||
XLMRobertaConfig,
|
||||
TFXLMRobertaForMaskedLM,
|
||||
XLMRobertaForMaskedLM,
|
||||
XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
),
|
||||
"transfo-xl": (
|
||||
TransfoXLConfig,
|
||||
TFTransfoXLLMHeadModel,
|
||||
@@ -263,13 +235,6 @@ MODEL_CLASSES = {
|
||||
ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
),
|
||||
"camembert": (
|
||||
CamembertConfig,
|
||||
TFCamembertForMaskedLM,
|
||||
CamembertForMaskedLM,
|
||||
CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
),
|
||||
"distilbert": (
|
||||
DistilBertConfig,
|
||||
TFDistilBertForMaskedLM,
|
||||
@@ -284,6 +249,13 @@ MODEL_CLASSES = {
|
||||
DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
),
|
||||
"distilbert-base-uncased-distilled-squad": (
|
||||
DistilBertConfig,
|
||||
TFDistilBertForQuestionAnswering,
|
||||
DistilBertForQuestionAnswering,
|
||||
DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
),
|
||||
"ctrl": (
|
||||
CTRLConfig,
|
||||
TFCTRLLMHeadModel,
|
||||
|
||||
@@ -152,8 +152,6 @@ class ModelCard(object):
|
||||
resolved_model_card_file = cached_path(
|
||||
model_card_file, cache_dir=cache_dir, force_download=True, proxies=proxies, resume_download=False
|
||||
)
|
||||
if resolved_model_card_file is None:
|
||||
raise EnvironmentError
|
||||
if resolved_model_card_file == model_card_file:
|
||||
logger.info("loading model card file {}".format(model_card_file))
|
||||
else:
|
||||
|
||||
@@ -698,7 +698,7 @@ class AlbertForSequenceClassification(AlbertPreTrainedModel):
|
||||
self.num_labels = config.num_labels
|
||||
|
||||
self.albert = AlbertModel(config)
|
||||
self.dropout = nn.Dropout(config.classifier_dropout_prob)
|
||||
self.dropout = nn.Dropout(config.hidden_dropout_prob)
|
||||
self.classifier = nn.Linear(config.hidden_size, self.config.num_labels)
|
||||
|
||||
self.init_weights()
|
||||
|
||||
@@ -25,7 +25,6 @@ from .configuration_auto import (
|
||||
CamembertConfig,
|
||||
CTRLConfig,
|
||||
DistilBertConfig,
|
||||
FlaubertConfig,
|
||||
GPT2Config,
|
||||
OpenAIGPTConfig,
|
||||
RobertaConfig,
|
||||
@@ -46,7 +45,6 @@ from .modeling_albert import (
|
||||
from .modeling_bert import (
|
||||
BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
BertForMaskedLM,
|
||||
BertForPreTraining,
|
||||
BertForQuestionAnswering,
|
||||
BertForSequenceClassification,
|
||||
BertForTokenClassification,
|
||||
@@ -68,19 +66,11 @@ from .modeling_distilbert import (
|
||||
DistilBertForTokenClassification,
|
||||
DistilBertModel,
|
||||
)
|
||||
from .modeling_flaubert import (
|
||||
FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
FlaubertForQuestionAnswering,
|
||||
FlaubertForSequenceClassification,
|
||||
FlaubertModel,
|
||||
FlaubertWithLMHeadModel,
|
||||
)
|
||||
from .modeling_gpt2 import GPT2_PRETRAINED_MODEL_ARCHIVE_MAP, GPT2LMHeadModel, GPT2Model
|
||||
from .modeling_openai import OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAP, OpenAIGPTLMHeadModel, OpenAIGPTModel
|
||||
from .modeling_roberta import (
|
||||
ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
RobertaForMaskedLM,
|
||||
RobertaForQuestionAnswering,
|
||||
RobertaForSequenceClassification,
|
||||
RobertaForTokenClassification,
|
||||
RobertaModel,
|
||||
@@ -129,7 +119,6 @@ ALL_PRETRAINED_MODEL_ARCHIVE_MAP = dict(
|
||||
ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
T5_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
]
|
||||
for key, value, in pretrained_map.items()
|
||||
@@ -148,31 +137,11 @@ MODEL_MAPPING = OrderedDict(
|
||||
(GPT2Config, GPT2Model),
|
||||
(TransfoXLConfig, TransfoXLModel),
|
||||
(XLNetConfig, XLNetModel),
|
||||
(FlaubertConfig, FlaubertModel),
|
||||
(XLMConfig, XLMModel),
|
||||
(CTRLConfig, CTRLModel),
|
||||
]
|
||||
)
|
||||
|
||||
MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
|
||||
[
|
||||
(T5Config, T5WithLMHeadModel),
|
||||
(DistilBertConfig, DistilBertForMaskedLM),
|
||||
(AlbertConfig, AlbertForMaskedLM),
|
||||
(CamembertConfig, CamembertForMaskedLM),
|
||||
(XLMRobertaConfig, XLMRobertaForMaskedLM),
|
||||
(RobertaConfig, RobertaForMaskedLM),
|
||||
(BertConfig, BertForPreTraining),
|
||||
(OpenAIGPTConfig, OpenAIGPTLMHeadModel),
|
||||
(GPT2Config, GPT2LMHeadModel),
|
||||
(TransfoXLConfig, TransfoXLLMHeadModel),
|
||||
(XLNetConfig, XLNetLMHeadModel),
|
||||
(FlaubertConfig, FlaubertWithLMHeadModel),
|
||||
(XLMConfig, XLMWithLMHeadModel),
|
||||
(CTRLConfig, CTRLLMHeadModel),
|
||||
]
|
||||
)
|
||||
|
||||
MODEL_WITH_LM_HEAD_MAPPING = OrderedDict(
|
||||
[
|
||||
(T5Config, T5WithLMHeadModel),
|
||||
@@ -186,7 +155,6 @@ MODEL_WITH_LM_HEAD_MAPPING = OrderedDict(
|
||||
(GPT2Config, GPT2LMHeadModel),
|
||||
(TransfoXLConfig, TransfoXLLMHeadModel),
|
||||
(XLNetConfig, XLNetLMHeadModel),
|
||||
(FlaubertConfig, FlaubertWithLMHeadModel),
|
||||
(XLMConfig, XLMWithLMHeadModel),
|
||||
(CTRLConfig, CTRLLMHeadModel),
|
||||
]
|
||||
@@ -201,7 +169,6 @@ MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING = OrderedDict(
|
||||
(RobertaConfig, RobertaForSequenceClassification),
|
||||
(BertConfig, BertForSequenceClassification),
|
||||
(XLNetConfig, XLNetForSequenceClassification),
|
||||
(FlaubertConfig, FlaubertForSequenceClassification),
|
||||
(XLMConfig, XLMForSequenceClassification),
|
||||
]
|
||||
)
|
||||
@@ -210,10 +177,8 @@ MODEL_FOR_QUESTION_ANSWERING_MAPPING = OrderedDict(
|
||||
[
|
||||
(DistilBertConfig, DistilBertForQuestionAnswering),
|
||||
(AlbertConfig, AlbertForQuestionAnswering),
|
||||
(RobertaConfig, RobertaForQuestionAnswering),
|
||||
(BertConfig, BertForQuestionAnswering),
|
||||
(XLNetConfig, XLNetForQuestionAnswering),
|
||||
(FlaubertConfig, FlaubertForQuestionAnswering),
|
||||
(XLMConfig, XLMForQuestionAnswering),
|
||||
]
|
||||
)
|
||||
@@ -265,7 +230,6 @@ class AutoModel(object):
|
||||
- isInstance of `transfo-xl` configuration class: :class:`~transformers.TransfoXLModel` (Transformer-XL model)
|
||||
- isInstance of `xlnet` configuration class: :class:`~transformers.XLNetModel` (XLNet model)
|
||||
- isInstance of `xlm` configuration class: :class:`~transformers.XLMModel` (XLM model)
|
||||
- isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertModel` (XLM model)
|
||||
|
||||
Examples::
|
||||
|
||||
@@ -306,7 +270,6 @@ class AutoModel(object):
|
||||
- contains `xlnet`: :class:`~transformers.XLNetModel` (XLNet model)
|
||||
- contains `xlm`: :class:`~transformers.XLMModel` (XLM model)
|
||||
- contains `ctrl`: :class:`~transformers.CTRLModel` (Salesforce CTRL model)
|
||||
- contains `flaubert`: :class:`~transformers.Flaubert` (Flaubert model)
|
||||
|
||||
The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
|
||||
To train the model, you should first set it back in training mode with `model.train()`
|
||||
@@ -383,158 +346,6 @@ class AutoModel(object):
|
||||
)
|
||||
|
||||
|
||||
class AutoModelForPreTraining(object):
|
||||
r"""
|
||||
:class:`~transformers.AutoModelForPreTraining` is a generic model class
|
||||
that will be instantiated as one of the model classes of the library -with the architecture used for pretraining this model– when created with the `AutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)`
|
||||
class method.
|
||||
|
||||
This class cannot be instantiated using `__init__()` (throws an error).
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
raise EnvironmentError(
|
||||
"AutoModelForPreTraining is designed to be instantiated "
|
||||
"using the `AutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)` or "
|
||||
"`AutoModelForPreTraining.from_config(config)` methods."
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_config(cls, config):
|
||||
r""" Instantiates one of the base model classes of the library
|
||||
from a configuration.
|
||||
|
||||
Args:
|
||||
config (:class:`~transformers.PretrainedConfig`):
|
||||
The model class to instantiate is selected based on the configuration class:
|
||||
|
||||
- isInstance of `distilbert` configuration class: :class:`~transformers.DistilBertModelForMaskedLM` (DistilBERT model)
|
||||
- isInstance of `roberta` configuration class: :class:`~transformers.RobertaModelForMaskedLM` (RoBERTa model)
|
||||
- isInstance of `bert` configuration class: :class:`~transformers.BertForPreTraining` (Bert model)
|
||||
- isInstance of `openai-gpt` configuration class: :class:`~transformers.OpenAIGPTLMHeadModel` (OpenAI GPT model)
|
||||
- isInstance of `gpt2` configuration class: :class:`~transformers.GPT2ModelLMHeadModel` (OpenAI GPT-2 model)
|
||||
- isInstance of `ctrl` configuration class: :class:`~transformers.CTRLModelLMHeadModel` (Salesforce CTRL model)
|
||||
- isInstance of `transfo-xl` configuration class: :class:`~transformers.TransfoXLLMHeadModel` (Transformer-XL model)
|
||||
- isInstance of `xlnet` configuration class: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
|
||||
- isInstance of `xlm` configuration class: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
|
||||
- isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertWithLMHeadModel` (Flaubert model)
|
||||
|
||||
Examples::
|
||||
|
||||
config = BertConfig.from_pretrained('bert-base-uncased') # Download configuration from S3 and cache.
|
||||
model = AutoModelForPreTraining.from_config(config) # E.g. model was saved using `save_pretrained('./test/saved_model/')`
|
||||
"""
|
||||
for config_class, model_class in MODEL_FOR_PRETRAINING_MAPPING.items():
|
||||
if isinstance(config, config_class):
|
||||
return model_class(config)
|
||||
raise ValueError(
|
||||
"Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
|
||||
"Model type should be one of {}.".format(
|
||||
config.__class__, cls.__name__, ", ".join(c.__name__ for c in MODEL_FOR_PRETRAINING_MAPPING.keys())
|
||||
)
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
|
||||
r""" Instantiates one of the model classes of the library -with the architecture used for pretraining this model– from a pre-trained model configuration.
|
||||
|
||||
The `from_pretrained()` method takes care of returning the correct model class instance
|
||||
based on the `model_type` property of the config object, or when it's missing,
|
||||
falling back to using pattern matching on the `pretrained_model_name_or_path` string.
|
||||
|
||||
The model class to instantiate is selected as the first pattern matching
|
||||
in the `pretrained_model_name_or_path` string (in the following order):
|
||||
- contains `t5`: :class:`~transformers.T5ModelWithLMHead` (T5 model)
|
||||
- contains `distilbert`: :class:`~transformers.DistilBertForMaskedLM` (DistilBERT model)
|
||||
- contains `albert`: :class:`~transformers.AlbertForMaskedLM` (ALBERT model)
|
||||
- contains `camembert`: :class:`~transformers.CamembertForMaskedLM` (CamemBERT model)
|
||||
- contains `xlm-roberta`: :class:`~transformers.XLMRobertaForMaskedLM` (XLM-RoBERTa model)
|
||||
- contains `roberta`: :class:`~transformers.RobertaForMaskedLM` (RoBERTa model)
|
||||
- contains `bert`: :class:`~transformers.BertForPreTraining` (Bert model)
|
||||
- contains `openai-gpt`: :class:`~transformers.OpenAIGPTLMHeadModel` (OpenAI GPT model)
|
||||
- contains `gpt2`: :class:`~transformers.GPT2LMHeadModel` (OpenAI GPT-2 model)
|
||||
- contains `transfo-xl`: :class:`~transformers.TransfoXLLMHeadModel` (Transformer-XL model)
|
||||
- contains `xlnet`: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
|
||||
- contains `xlm`: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
|
||||
- contains `ctrl`: :class:`~transformers.CTRLLMHeadModel` (Salesforce CTRL model)
|
||||
- contains `flaubert`: :class:`~transformers.FlaubertWithLMHeadModel` (Flaubert model)
|
||||
|
||||
The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
|
||||
To train the model, you should first set it back in training mode with `model.train()`
|
||||
|
||||
Args:
|
||||
pretrained_model_name_or_path:
|
||||
Either:
|
||||
|
||||
- a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
|
||||
- a string with the `identifier name` of a pre-trained model that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``.
|
||||
- a path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g.: ``./my_model_directory/``.
|
||||
- a path or url to a `tensorflow index checkpoint file` (e.g. `./tf_model/model.ckpt.index`). In this case, ``from_tf`` should be set to True and a configuration object should be provided as ``config`` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
|
||||
model_args: (`optional`) Sequence of positional arguments:
|
||||
All remaning positional arguments will be passed to the underlying model's ``__init__`` method
|
||||
config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
|
||||
Configuration for the model to use instead of an automatically loaded configuation. Configuration can be automatically loaded when:
|
||||
|
||||
- the model is a model provided by the library (loaded with the ``shortcut-name`` string of a pretrained model), or
|
||||
- the model was saved using :func:`~transformers.PreTrainedModel.save_pretrained` and is reloaded by suppling the save directory.
|
||||
- the model is loaded by suppling a local directory as ``pretrained_model_name_or_path`` and a configuration JSON file named `config.json` is found in the directory.
|
||||
|
||||
state_dict: (`optional`) dict:
|
||||
an optional state dictionnary for the model to use instead of a state dictionary loaded from saved weights file.
|
||||
This option can be used if you want to create a model from a pretrained configuration but load your own weights.
|
||||
In this case though, you should check if using :func:`~transformers.PreTrainedModel.save_pretrained` and :func:`~transformers.PreTrainedModel.from_pretrained` is not a simpler option.
|
||||
cache_dir: (`optional`) string:
|
||||
Path to a directory in which a downloaded pre-trained model
|
||||
configuration should be cached if the standard cache should not be used.
|
||||
force_download: (`optional`) boolean, default False:
|
||||
Force to (re-)download the model weights and configuration files and override the cached versions if they exists.
|
||||
resume_download: (`optional`) boolean, default False:
|
||||
Do not delete incompletely received file. Attempt to resume the download if such a file exists.
|
||||
proxies: (`optional`) dict, default None:
|
||||
A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.
|
||||
The proxies are used on each request.
|
||||
output_loading_info: (`optional`) boolean:
|
||||
Set to ``True`` to also return a dictionnary containing missing keys, unexpected keys and error messages.
|
||||
kwargs: (`optional`) Remaining dictionary of keyword arguments:
|
||||
Can be used to update the configuration object (after it being loaded) and initiate the model.
|
||||
(e.g. ``output_attention=True``). Behave differently depending on whether a `config` is provided or
|
||||
automatically loaded:
|
||||
|
||||
- If a configuration is provided with ``config``, ``**kwargs`` will be directly passed to the
|
||||
underlying model's ``__init__`` method (we assume all relevant updates to the configuration have
|
||||
already been done)
|
||||
- If a configuration is not provided, ``kwargs`` will be first passed to the configuration class
|
||||
initialization function (:func:`~transformers.PretrainedConfig.from_pretrained`). Each key of
|
||||
``kwargs`` that corresponds to a configuration attribute will be used to override said attribute
|
||||
with the supplied ``kwargs`` value. Remaining keys that do not correspond to any configuration
|
||||
attribute will be passed to the underlying model's ``__init__`` function.
|
||||
|
||||
Examples::
|
||||
|
||||
model = AutoModelForPreTraining.from_pretrained('bert-base-uncased') # Download model and configuration from S3 and cache.
|
||||
model = AutoModelForPreTraining.from_pretrained('./test/bert_model/') # E.g. model was saved using `save_pretrained('./test/saved_model/')`
|
||||
model = AutoModelForPreTraining.from_pretrained('bert-base-uncased', output_attention=True) # Update configuration during loading
|
||||
assert model.config.output_attention == True
|
||||
# Loading from a TF checkpoint file instead of a PyTorch model (slower)
|
||||
config = AutoConfig.from_json_file('./tf_model/bert_tf_model_config.json')
|
||||
model = AutoModelForPreTraining.from_pretrained('./tf_model/bert_tf_checkpoint.ckpt.index', from_tf=True, config=config)
|
||||
|
||||
"""
|
||||
config = kwargs.pop("config", None)
|
||||
if not isinstance(config, PretrainedConfig):
|
||||
config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
|
||||
|
||||
for config_class, model_class in MODEL_FOR_PRETRAINING_MAPPING.items():
|
||||
if isinstance(config, config_class):
|
||||
return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
|
||||
raise ValueError(
|
||||
"Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
|
||||
"Model type should be one of {}.".format(
|
||||
config.__class__, cls.__name__, ", ".join(c.__name__ for c in MODEL_FOR_PRETRAINING_MAPPING.keys())
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class AutoModelWithLMHead(object):
|
||||
r"""
|
||||
:class:`~transformers.AutoModelWithLMHead` is a generic model class
|
||||
@@ -570,7 +381,6 @@ class AutoModelWithLMHead(object):
|
||||
- isInstance of `transfo-xl` configuration class: :class:`~transformers.TransfoXLLMHeadModel` (Transformer-XL model)
|
||||
- isInstance of `xlnet` configuration class: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
|
||||
- isInstance of `xlm` configuration class: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
|
||||
- isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertWithLMHeadModel` (Flaubert model)
|
||||
|
||||
Examples::
|
||||
|
||||
@@ -611,7 +421,6 @@ class AutoModelWithLMHead(object):
|
||||
- contains `xlnet`: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
|
||||
- contains `xlm`: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
|
||||
- contains `ctrl`: :class:`~transformers.CTRLLMHeadModel` (Salesforce CTRL model)
|
||||
- contains `flaubert`: :class:`~transformers.FlaubertWithLMHeadModel` (Flaubert model)
|
||||
|
||||
The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
|
||||
To train the model, you should first set it back in training mode with `model.train()`
|
||||
@@ -723,7 +532,6 @@ class AutoModelForSequenceClassification(object):
|
||||
- isInstance of `bert` configuration class: :class:`~transformers.BertModelForSequenceClassification` (Bert model)
|
||||
- isInstance of `xlnet` configuration class: :class:`~transformers.XLNetModelForSequenceClassification` (XLNet model)
|
||||
- isInstance of `xlm` configuration class: :class:`~transformers.XLMModelForSequenceClassification` (XLM model)
|
||||
- isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertForSequenceClassification` (Flaubert model)
|
||||
|
||||
|
||||
Examples::
|
||||
@@ -761,7 +569,7 @@ class AutoModelForSequenceClassification(object):
|
||||
- contains `roberta`: :class:`~transformers.RobertaForSequenceClassification` (RoBERTa model)
|
||||
- contains `bert`: :class:`~transformers.BertForSequenceClassification` (Bert model)
|
||||
- contains `xlnet`: :class:`~transformers.XLNetForSequenceClassification` (XLNet model)
|
||||
- contains `flaubert`: :class:`~transformers.FlaubertForSequenceClassification` (Flaubert model)
|
||||
- contains `xlm`: :class:`~transformers.XLMForSequenceClassification` (XLM model)
|
||||
|
||||
The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
|
||||
To train the model, you should first set it back in training mode with `model.train()`
|
||||
@@ -871,7 +679,6 @@ class AutoModelForQuestionAnswering(object):
|
||||
- isInstance of `bert` configuration class: :class:`~transformers.BertModelForQuestionAnswering` (Bert model)
|
||||
- isInstance of `xlnet` configuration class: :class:`~transformers.XLNetModelForQuestionAnswering` (XLNet model)
|
||||
- isInstance of `xlm` configuration class: :class:`~transformers.XLMModelForQuestionAnswering` (XLM model)
|
||||
- isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertForQuestionAnswering` (XLM model)
|
||||
|
||||
Examples::
|
||||
|
||||
@@ -907,7 +714,6 @@ class AutoModelForQuestionAnswering(object):
|
||||
- contains `bert`: :class:`~transformers.BertForQuestionAnswering` (Bert model)
|
||||
- contains `xlnet`: :class:`~transformers.XLNetForQuestionAnswering` (XLNet model)
|
||||
- contains `xlm`: :class:`~transformers.XLMForQuestionAnswering` (XLM model)
|
||||
- contains `flaubert`: :class:`~transformers.FlaubertForQuestionAnswering` (XLM model)
|
||||
|
||||
The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
|
||||
To train the model, you should first set it back in training mode with `model.train()`
|
||||
|
||||
@@ -53,7 +53,6 @@ BERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
|
||||
"bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-pytorch_model.bin",
|
||||
"bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/pytorch_model.bin",
|
||||
"bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/pytorch_model.bin",
|
||||
"bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/pytorch_model.bin",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -33,8 +33,6 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
|
||||
"camembert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/camembert-base-pytorch_model.bin",
|
||||
"umberto-commoncrawl-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/Musixmatch/umberto-commoncrawl-cased-v1/pytorch_model.bin",
|
||||
"umberto-wikipedia-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/Musixmatch/umberto-wikipedia-uncased-v1/pytorch_model.bin",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -30,7 +30,9 @@ from .modeling_utils import Conv1D, PreTrainedModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CTRL_PRETRAINED_MODEL_ARCHIVE_MAP = {"ctrl": "https://storage.googleapis.com/sf-ctrl/pytorch/seqlen256_v1.bin"}
|
||||
CTRL_PRETRAINED_MODEL_ARCHIVE_MAP = {
|
||||
"ctrl": "https://s3.amazonaws.com/models.huggingface.co/bert/ctrl-pytorch_model.bin"
|
||||
}
|
||||
|
||||
|
||||
def angle_defn(pos, i, d_model_size):
|
||||
|
||||
@@ -232,7 +232,7 @@ class PreTrainedEncoderDecoder(nn.Module):
|
||||
encoder_outputs = ()
|
||||
|
||||
kwargs_decoder["encoder_hidden_states"] = encoder_hidden_states
|
||||
decoder_outputs = self.decoder(decoder_input_ids, **kwargs_decoder)
|
||||
decoder_outputs = self.decoder(decoder_input_ids, encoder_hidden_states, **kwargs_decoder)
|
||||
|
||||
return decoder_outputs + encoder_outputs
|
||||
|
||||
|
||||
@@ -1,385 +0,0 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2019-present CNRS, Facebook Inc. and the HuggingFace Inc. team.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
""" PyTorch Flaubert model, based on XLM. """
|
||||
|
||||
|
||||
import logging
|
||||
import random
|
||||
|
||||
import torch
|
||||
from torch.nn import functional as F
|
||||
|
||||
from .configuration_flaubert import FlaubertConfig
|
||||
from .file_utils import add_start_docstrings, add_start_docstrings_to_callable
|
||||
from .modeling_xlm import (
|
||||
XLMForQuestionAnswering,
|
||||
XLMForQuestionAnsweringSimple,
|
||||
XLMForSequenceClassification,
|
||||
XLMModel,
|
||||
XLMWithLMHeadModel,
|
||||
get_masks,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
|
||||
"flaubert-small-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_small_cased/pytorch_model.bin",
|
||||
"flaubert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/pytorch_model.bin",
|
||||
"flaubert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_cased/pytorch_model.bin",
|
||||
"flaubert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_large_cased/pytorch_model.bin",
|
||||
}
|
||||
|
||||
|
||||
FLAUBERT_START_DOCSTRING = r"""
|
||||
|
||||
This model is a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`_ sub-class.
|
||||
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general
|
||||
usage and behavior.
|
||||
|
||||
Parameters:
|
||||
config (:class:`~transformers.FlaubertConfig`): Model configuration class with all the parameters of the model.
|
||||
Initializing with a config file does not load the weights associated with the model, only the configuration.
|
||||
Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights.
|
||||
"""
|
||||
|
||||
FLAUBERT_INPUTS_DOCSTRING = r"""
|
||||
Args:
|
||||
input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`):
|
||||
Indices of input sequence tokens in the vocabulary.
|
||||
|
||||
Indices can be obtained using :class:`transformers.BertTokenizer`.
|
||||
See :func:`transformers.PreTrainedTokenizer.encode` and
|
||||
:func:`transformers.PreTrainedTokenizer.encode_plus` for details.
|
||||
|
||||
`What are input IDs? <../glossary.html#input-ids>`__
|
||||
attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
|
||||
Mask to avoid performing attention on padding token indices.
|
||||
Mask values selected in ``[0, 1]``:
|
||||
``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
|
||||
|
||||
`What are attention masks? <../glossary.html#attention-mask>`__
|
||||
token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
|
||||
Segment token indices to indicate first and second portions of the inputs.
|
||||
Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1``
|
||||
corresponds to a `sentence B` token
|
||||
|
||||
`What are token type IDs? <../glossary.html#token-type-ids>`_
|
||||
position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
|
||||
Indices of positions of each input sequence tokens in the position embeddings.
|
||||
Selected in the range ``[0, config.max_position_embeddings - 1]``.
|
||||
|
||||
`What are position IDs? <../glossary.html#position-ids>`_
|
||||
lengths (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`):
|
||||
Length of each sentence that can be used to avoid performing attention on padding token indices.
|
||||
You can also use `attention_mask` for the same result (see above), kept here for compatbility.
|
||||
Indices selected in ``[0, ..., input_ids.size(-1)]``:
|
||||
cache (:obj:`Dict[str, torch.FloatTensor]`, `optional`, defaults to :obj:`None`):
|
||||
dictionary with ``torch.FloatTensor`` that contains pre-computed
|
||||
hidden-states (key and values in the attention blocks) as computed by the model
|
||||
(see `cache` output below). Can be used to speed up sequential decoding.
|
||||
The dictionary object will be modified in-place during the forward pass to add newly computed hidden-states.
|
||||
head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`):
|
||||
Mask to nullify selected heads of the self-attention modules.
|
||||
Mask values selected in ``[0, 1]``:
|
||||
:obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**.
|
||||
input_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
|
||||
Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation.
|
||||
This is useful if you want more control over how to convert `input_ids` indices into associated vectors
|
||||
than the model's internal embedding lookup matrix.
|
||||
"""
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"The bare Flaubert Model transformer outputting raw hidden-states without any specific head on top.",
|
||||
FLAUBERT_START_DOCSTRING,
|
||||
)
|
||||
class FlaubertModel(XLMModel):
|
||||
|
||||
config_class = FlaubertConfig
|
||||
pretrained_model_archive_map = FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
def __init__(self, config): # , dico, is_encoder, with_output):
|
||||
super(FlaubertModel, self).__init__(config)
|
||||
self.layerdrop = getattr(config, "layerdrop", 0.0)
|
||||
self.pre_norm = getattr(config, "pre_norm", False)
|
||||
|
||||
@add_start_docstrings_to_callable(FLAUBERT_INPUTS_DOCSTRING)
|
||||
def forward(
|
||||
self,
|
||||
input_ids=None,
|
||||
attention_mask=None,
|
||||
langs=None,
|
||||
token_type_ids=None,
|
||||
position_ids=None,
|
||||
lengths=None,
|
||||
cache=None,
|
||||
head_mask=None,
|
||||
inputs_embeds=None,
|
||||
):
|
||||
r"""
|
||||
Return:
|
||||
:obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs:
|
||||
last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
|
||||
Sequence of hidden-states at the output of the last layer of the model.
|
||||
hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``):
|
||||
Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
|
||||
of shape :obj:`(batch_size, sequence_length, hidden_size)`.
|
||||
|
||||
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
|
||||
attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
|
||||
Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
|
||||
:obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
|
||||
|
||||
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
|
||||
heads.
|
||||
|
||||
Examples::
|
||||
|
||||
tokenizer = FlaubertTokenizer.from_pretrained('flaubert-base-cased')
|
||||
model = FlaubertModel.from_pretrained('flaubert-base-cased')
|
||||
input_ids = torch.tensor(tokenizer.encode("Le chat manges une pomme.", add_special_tokens=True)).unsqueeze(0) # Batch size 1
|
||||
outputs = model(input_ids)
|
||||
last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple
|
||||
|
||||
"""
|
||||
# removed: src_enc=None, src_len=None
|
||||
if input_ids is not None:
|
||||
bs, slen = input_ids.size()
|
||||
else:
|
||||
bs, slen = inputs_embeds.size()[:-1]
|
||||
|
||||
if lengths is None:
|
||||
if input_ids is not None:
|
||||
lengths = (input_ids != self.pad_index).sum(dim=1).long()
|
||||
else:
|
||||
lengths = torch.LongTensor([slen] * bs)
|
||||
# mask = input_ids != self.pad_index
|
||||
|
||||
# check inputs
|
||||
assert lengths.size(0) == bs
|
||||
assert lengths.max().item() <= slen
|
||||
# input_ids = input_ids.transpose(0, 1) # batch size as dimension 0
|
||||
# assert (src_enc is None) == (src_len is None)
|
||||
# if src_enc is not None:
|
||||
# assert self.is_decoder
|
||||
# assert src_enc.size(0) == bs
|
||||
|
||||
# generate masks
|
||||
mask, attn_mask = get_masks(slen, lengths, self.causal, padding_mask=attention_mask)
|
||||
# if self.is_decoder and src_enc is not None:
|
||||
# src_mask = torch.arange(src_len.max(), dtype=torch.long, device=lengths.device) < src_len[:, None]
|
||||
|
||||
device = input_ids.device if input_ids is not None else inputs_embeds.device
|
||||
|
||||
# position_ids
|
||||
if position_ids is None:
|
||||
position_ids = torch.arange(slen, dtype=torch.long, device=device)
|
||||
position_ids = position_ids.unsqueeze(0).expand((bs, slen))
|
||||
else:
|
||||
assert position_ids.size() == (bs, slen) # (slen, bs)
|
||||
# position_ids = position_ids.transpose(0, 1)
|
||||
|
||||
# langs
|
||||
if langs is not None:
|
||||
assert langs.size() == (bs, slen) # (slen, bs)
|
||||
# langs = langs.transpose(0, 1)
|
||||
|
||||
# Prepare head mask if needed
|
||||
# 1.0 in head_mask indicate we keep the head
|
||||
# attention_probs has shape bsz x n_heads x N x N
|
||||
# input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
|
||||
# and head_mask is converted to shape [num_hidden_layers x batch x num_heads x qlen x klen]
|
||||
if head_mask is not None:
|
||||
if head_mask.dim() == 1:
|
||||
head_mask = head_mask.unsqueeze(0).unsqueeze(0).unsqueeze(-1).unsqueeze(-1)
|
||||
head_mask = head_mask.expand(self.n_layers, -1, -1, -1, -1)
|
||||
elif head_mask.dim() == 2:
|
||||
head_mask = (
|
||||
head_mask.unsqueeze(1).unsqueeze(-1).unsqueeze(-1)
|
||||
) # We can specify head_mask for each layer
|
||||
head_mask = head_mask.to(
|
||||
dtype=next(self.parameters()).dtype
|
||||
) # switch to fload if need + fp16 compatibility
|
||||
else:
|
||||
head_mask = [None] * self.n_layers
|
||||
|
||||
# do not recompute cached elements
|
||||
if cache is not None and input_ids is not None:
|
||||
_slen = slen - cache["slen"]
|
||||
input_ids = input_ids[:, -_slen:]
|
||||
position_ids = position_ids[:, -_slen:]
|
||||
if langs is not None:
|
||||
langs = langs[:, -_slen:]
|
||||
mask = mask[:, -_slen:]
|
||||
attn_mask = attn_mask[:, -_slen:]
|
||||
|
||||
# embeddings
|
||||
if inputs_embeds is None:
|
||||
inputs_embeds = self.embeddings(input_ids)
|
||||
|
||||
tensor = inputs_embeds + self.position_embeddings(position_ids).expand_as(inputs_embeds)
|
||||
if langs is not None and self.use_lang_emb:
|
||||
tensor = tensor + self.lang_embeddings(langs)
|
||||
if token_type_ids is not None:
|
||||
tensor = tensor + self.embeddings(token_type_ids)
|
||||
tensor = self.layer_norm_emb(tensor)
|
||||
tensor = F.dropout(tensor, p=self.dropout, training=self.training)
|
||||
tensor *= mask.unsqueeze(-1).to(tensor.dtype)
|
||||
|
||||
# transformer layers
|
||||
hidden_states = ()
|
||||
attentions = ()
|
||||
for i in range(self.n_layers):
|
||||
# LayerDrop
|
||||
dropout_probability = random.uniform(0, 1)
|
||||
if self.training and (dropout_probability < self.layerdrop):
|
||||
continue
|
||||
|
||||
if self.output_hidden_states:
|
||||
hidden_states = hidden_states + (tensor,)
|
||||
|
||||
# self attention
|
||||
if not self.pre_norm:
|
||||
attn_outputs = self.attentions[i](tensor, attn_mask, cache=cache, head_mask=head_mask[i])
|
||||
attn = attn_outputs[0]
|
||||
if self.output_attentions:
|
||||
attentions = attentions + (attn_outputs[1],)
|
||||
attn = F.dropout(attn, p=self.dropout, training=self.training)
|
||||
tensor = tensor + attn
|
||||
tensor = self.layer_norm1[i](tensor)
|
||||
else:
|
||||
tensor_normalized = self.layer_norm1[i](tensor)
|
||||
attn_outputs = self.attentions[i](tensor_normalized, attn_mask, cache=cache, head_mask=head_mask[i])
|
||||
attn = attn_outputs[0]
|
||||
if self.output_attentions:
|
||||
attentions = attentions + (attn_outputs[1],)
|
||||
attn = F.dropout(attn, p=self.dropout, training=self.training)
|
||||
tensor = tensor + attn
|
||||
|
||||
# encoder attention (for decoder only)
|
||||
# if self.is_decoder and src_enc is not None:
|
||||
# attn = self.encoder_attn[i](tensor, src_mask, kv=src_enc, cache=cache)
|
||||
# attn = F.dropout(attn, p=self.dropout, training=self.training)
|
||||
# tensor = tensor + attn
|
||||
# tensor = self.layer_norm15[i](tensor)
|
||||
|
||||
# FFN
|
||||
if not self.pre_norm:
|
||||
tensor = tensor + self.ffns[i](tensor)
|
||||
tensor = self.layer_norm2[i](tensor)
|
||||
else:
|
||||
tensor_normalized = self.layer_norm2[i](tensor)
|
||||
tensor = tensor + self.ffns[i](tensor_normalized)
|
||||
|
||||
tensor *= mask.unsqueeze(-1).to(tensor.dtype)
|
||||
|
||||
# Add last hidden state
|
||||
if self.output_hidden_states:
|
||||
hidden_states = hidden_states + (tensor,)
|
||||
|
||||
# update cache length
|
||||
if cache is not None:
|
||||
cache["slen"] += tensor.size(1)
|
||||
|
||||
# move back sequence length to dimension 0
|
||||
# tensor = tensor.transpose(0, 1)
|
||||
|
||||
outputs = (tensor,)
|
||||
if self.output_hidden_states:
|
||||
outputs = outputs + (hidden_states,)
|
||||
if self.output_attentions:
|
||||
outputs = outputs + (attentions,)
|
||||
return outputs # outputs, (hidden_states), (attentions)
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"""The Flaubert Model transformer with a language modeling head on top
|
||||
(linear layer with weights tied to the input embeddings). """,
|
||||
FLAUBERT_START_DOCSTRING,
|
||||
)
|
||||
class FlaubertWithLMHeadModel(XLMWithLMHeadModel):
|
||||
"""
|
||||
This class overrides :class:`~transformers.XLMWithLMHeadModel`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = FlaubertConfig
|
||||
pretrained_model_archive_map = FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
def __init__(self, config):
|
||||
super(FlaubertWithLMHeadModel, self).__init__(config)
|
||||
self.transformer = FlaubertModel(config)
|
||||
self.init_weights()
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"""Flaubert Model with a sequence classification/regression head on top (a linear layer on top of
|
||||
the pooled output) e.g. for GLUE tasks. """,
|
||||
FLAUBERT_START_DOCSTRING,
|
||||
)
|
||||
class FlaubertForSequenceClassification(XLMForSequenceClassification):
|
||||
"""
|
||||
This class overrides :class:`~transformers.XLMForSequenceClassification`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = FlaubertConfig
|
||||
pretrained_model_archive_map = FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
def __init__(self, config):
|
||||
super(FlaubertForSequenceClassification, self).__init__(config)
|
||||
self.transformer = FlaubertModel(config)
|
||||
self.init_weights()
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"""Flaubert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of
|
||||
the hidden-states output to compute `span start logits` and `span end logits`). """,
|
||||
FLAUBERT_START_DOCSTRING,
|
||||
)
|
||||
class FlaubertForQuestionAnsweringSimple(XLMForQuestionAnsweringSimple):
|
||||
"""
|
||||
This class overrides :class:`~transformers.XLMForQuestionAnsweringSimple`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = FlaubertConfig
|
||||
pretrained_model_archive_map = FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
def __init__(self, config):
|
||||
super(FlaubertForQuestionAnsweringSimple, self).__init__(config)
|
||||
self.transformer = FlaubertModel(config)
|
||||
self.init_weights()
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"""Flaubert Model with a beam-search span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of
|
||||
the hidden-states output to compute `span start logits` and `span end logits`). """,
|
||||
FLAUBERT_START_DOCSTRING,
|
||||
)
|
||||
class FlaubertForQuestionAnswering(XLMForQuestionAnswering):
|
||||
"""
|
||||
This class overrides :class:`~transformers.XLMForQuestionAnswering`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = FlaubertConfig
|
||||
pretrained_model_archive_map = FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
def __init__(self, config):
|
||||
super(FlaubertForQuestionAnswering, self).__init__(config)
|
||||
self.transformer = FlaubertModel(config)
|
||||
self.init_weights()
|
||||
@@ -42,7 +42,6 @@ from .modeling_tf_albert import (
|
||||
from .modeling_tf_bert import (
|
||||
TF_BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
|
||||
TFBertForMaskedLM,
|
||||
TFBertForPreTraining,
|
||||
TFBertForQuestionAnswering,
|
||||
TFBertForSequenceClassification,
|
||||
TFBertForTokenClassification,
|
||||
@@ -126,22 +125,6 @@ TF_MODEL_MAPPING = OrderedDict(
|
||||
]
|
||||
)
|
||||
|
||||
TF_MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
|
||||
[
|
||||
(T5Config, TFT5WithLMHeadModel),
|
||||
(DistilBertConfig, TFDistilBertForMaskedLM),
|
||||
(AlbertConfig, TFAlbertForMaskedLM),
|
||||
(RobertaConfig, TFRobertaForMaskedLM),
|
||||
(BertConfig, TFBertForPreTraining),
|
||||
(OpenAIGPTConfig, TFOpenAIGPTLMHeadModel),
|
||||
(GPT2Config, TFGPT2LMHeadModel),
|
||||
(TransfoXLConfig, TFTransfoXLLMHeadModel),
|
||||
(XLNetConfig, TFXLNetLMHeadModel),
|
||||
(XLMConfig, TFXLMWithLMHeadModel),
|
||||
(CTRLConfig, TFCTRLLMHeadModel),
|
||||
]
|
||||
)
|
||||
|
||||
TF_MODEL_WITH_LM_HEAD_MAPPING = OrderedDict(
|
||||
[
|
||||
(T5Config, TFT5WithLMHeadModel),
|
||||
@@ -346,154 +329,6 @@ class TFAutoModel(object):
|
||||
)
|
||||
|
||||
|
||||
class TFAutoModelForPreTraining(object):
|
||||
r"""
|
||||
:class:`~transformers.TFAutoModelForPreTraining` is a generic model class
|
||||
that will be instantiated as one of the model classes of the library -with the architecture used for pretraining this model– when created with the `TFAutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)`
|
||||
class method.
|
||||
|
||||
This class cannot be instantiated using `__init__()` (throws an error).
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
raise EnvironmentError(
|
||||
"TFAutoModelForPreTraining is designed to be instantiated "
|
||||
"using the `TFAutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)` or "
|
||||
"`TFAutoModelForPreTraining.from_config(config)` methods."
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_config(cls, config):
|
||||
r""" Instantiates one of the base model classes of the library
|
||||
from a configuration.
|
||||
|
||||
Args:
|
||||
config (:class:`~transformers.PretrainedConfig`):
|
||||
The model class to instantiate is selected based on the configuration class:
|
||||
|
||||
- isInstance of `distilbert` configuration class: :class:`~transformers.TFDistilBertModelForMaskedLM` (DistilBERT model)
|
||||
- isInstance of `roberta` configuration class: :class:`~transformers.TFRobertaModelForMaskedLM` (RoBERTa model)
|
||||
- isInstance of `bert` configuration class: :class:`~transformers.TFBertForPreTraining` (Bert model)
|
||||
- isInstance of `openai-gpt` configuration class: :class:`~transformers.TFOpenAIGPTLMHeadModel` (OpenAI GPT model)
|
||||
- isInstance of `gpt2` configuration class: :class:`~transformers.TFGPT2ModelLMHeadModel` (OpenAI GPT-2 model)
|
||||
- isInstance of `ctrl` configuration class: :class:`~transformers.TFCTRLModelLMHeadModel` (Salesforce CTRL model)
|
||||
- isInstance of `transfo-xl` configuration class: :class:`~transformers.TFTransfoXLLMHeadModel` (Transformer-XL model)
|
||||
- isInstance of `xlnet` configuration class: :class:`~transformers.TFXLNetLMHeadModel` (XLNet model)
|
||||
- isInstance of `xlm` configuration class: :class:`~transformers.TFXLMWithLMHeadModel` (XLM model)
|
||||
|
||||
Examples::
|
||||
|
||||
config = BertConfig.from_pretrained('bert-base-uncased') # Download configuration from S3 and cache.
|
||||
model = TFAutoModelForPreTraining.from_config(config) # E.g. model was saved using `save_pretrained('./test/saved_model/')`
|
||||
"""
|
||||
for config_class, model_class in TF_MODEL_FOR_PRETRAINING_MAPPING.items():
|
||||
if isinstance(config, config_class):
|
||||
return model_class(config)
|
||||
raise ValueError(
|
||||
"Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
|
||||
"Model type should be one of {}.".format(
|
||||
config.__class__, cls.__name__, ", ".join(c.__name__ for c in TF_MODEL_FOR_PRETRAINING_MAPPING.keys())
|
||||
)
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
|
||||
r""" Instantiates one of the model classes of the library -with the architecture used for pretraining this model– from a pre-trained model configuration.
|
||||
|
||||
The `from_pretrained()` method takes care of returning the correct model class instance
|
||||
based on the `model_type` property of the config object, or when it's missing,
|
||||
falling back to using pattern matching on the `pretrained_model_name_or_path` string.
|
||||
|
||||
The model class to instantiate is selected as the first pattern matching
|
||||
in the `pretrained_model_name_or_path` string (in the following order):
|
||||
- contains `t5`: :class:`~transformers.TFT5ModelWithLMHead` (T5 model)
|
||||
- contains `distilbert`: :class:`~transformers.TFDistilBertForMaskedLM` (DistilBERT model)
|
||||
- contains `albert`: :class:`~transformers.TFAlbertForMaskedLM` (ALBERT model)
|
||||
- contains `roberta`: :class:`~transformers.TFRobertaForMaskedLM` (RoBERTa model)
|
||||
- contains `bert`: :class:`~transformers.TFBertForPreTraining` (Bert model)
|
||||
- contains `openai-gpt`: :class:`~transformers.TFOpenAIGPTLMHeadModel` (OpenAI GPT model)
|
||||
- contains `gpt2`: :class:`~transformers.TFGPT2LMHeadModel` (OpenAI GPT-2 model)
|
||||
- contains `transfo-xl`: :class:`~transformers.TFTransfoXLLMHeadModel` (Transformer-XL model)
|
||||
- contains `xlnet`: :class:`~transformers.TFXLNetLMHeadModel` (XLNet model)
|
||||
- contains `xlm`: :class:`~transformers.TFXLMWithLMHeadModel` (XLM model)
|
||||
- contains `ctrl`: :class:`~transformers.TFCTRLLMHeadModel` (Salesforce CTRL model)
|
||||
|
||||
The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
|
||||
To train the model, you should first set it back in training mode with `model.train()`
|
||||
|
||||
Args:
|
||||
pretrained_model_name_or_path:
|
||||
Either:
|
||||
|
||||
- a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
|
||||
- a string with the `identifier name` of a pre-trained model that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``.
|
||||
- a path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g.: ``./my_model_directory/``.
|
||||
- a path or url to a `tensorflow index checkpoint file` (e.g. `./tf_model/model.ckpt.index`). In this case, ``from_tf`` should be set to True and a configuration object should be provided as ``config`` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
|
||||
model_args: (`optional`) Sequence of positional arguments:
|
||||
All remaning positional arguments will be passed to the underlying model's ``__init__`` method
|
||||
config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
|
||||
Configuration for the model to use instead of an automatically loaded configuation. Configuration can be automatically loaded when:
|
||||
|
||||
- the model is a model provided by the library (loaded with the ``shortcut-name`` string of a pretrained model), or
|
||||
- the model was saved using :func:`~transformers.PreTrainedModel.save_pretrained` and is reloaded by suppling the save directory.
|
||||
- the model is loaded by suppling a local directory as ``pretrained_model_name_or_path`` and a configuration JSON file named `config.json` is found in the directory.
|
||||
|
||||
state_dict: (`optional`) dict:
|
||||
an optional state dictionnary for the model to use instead of a state dictionary loaded from saved weights file.
|
||||
This option can be used if you want to create a model from a pretrained configuration but load your own weights.
|
||||
In this case though, you should check if using :func:`~transformers.PreTrainedModel.save_pretrained` and :func:`~transformers.PreTrainedModel.from_pretrained` is not a simpler option.
|
||||
cache_dir: (`optional`) string:
|
||||
Path to a directory in which a downloaded pre-trained model
|
||||
configuration should be cached if the standard cache should not be used.
|
||||
force_download: (`optional`) boolean, default False:
|
||||
Force to (re-)download the model weights and configuration files and override the cached versions if they exists.
|
||||
resume_download: (`optional`) boolean, default False:
|
||||
Do not delete incompletely received file. Attempt to resume the download if such a file exists.
|
||||
proxies: (`optional`) dict, default None:
|
||||
A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.
|
||||
The proxies are used on each request.
|
||||
output_loading_info: (`optional`) boolean:
|
||||
Set to ``True`` to also return a dictionnary containing missing keys, unexpected keys and error messages.
|
||||
kwargs: (`optional`) Remaining dictionary of keyword arguments:
|
||||
Can be used to update the configuration object (after it being loaded) and initiate the model.
|
||||
(e.g. ``output_attention=True``). Behave differently depending on whether a `config` is provided or
|
||||
automatically loaded:
|
||||
|
||||
- If a configuration is provided with ``config``, ``**kwargs`` will be directly passed to the
|
||||
underlying model's ``__init__`` method (we assume all relevant updates to the configuration have
|
||||
already been done)
|
||||
- If a configuration is not provided, ``kwargs`` will be first passed to the configuration class
|
||||
initialization function (:func:`~transformers.PretrainedConfig.from_pretrained`). Each key of
|
||||
``kwargs`` that corresponds to a configuration attribute will be used to override said attribute
|
||||
with the supplied ``kwargs`` value. Remaining keys that do not correspond to any configuration
|
||||
attribute will be passed to the underlying model's ``__init__`` function.
|
||||
|
||||
Examples::
|
||||
|
||||
model = TFAutoModelForPreTraining.from_pretrained('bert-base-uncased') # Download model and configuration from S3 and cache.
|
||||
model = TFAutoModelForPreTraining.from_pretrained('./test/bert_model/') # E.g. model was saved using `save_pretrained('./test/saved_model/')`
|
||||
model = TFAutoModelForPreTraining.from_pretrained('bert-base-uncased', output_attention=True) # Update configuration during loading
|
||||
assert model.config.output_attention == True
|
||||
# Loading from a TF checkpoint file instead of a PyTorch model (slower)
|
||||
config = AutoConfig.from_json_file('./tf_model/bert_tf_model_config.json')
|
||||
model = TFAutoModelForPreTraining.from_pretrained('./tf_model/bert_tf_checkpoint.ckpt.index', from_tf=True, config=config)
|
||||
|
||||
"""
|
||||
config = kwargs.pop("config", None)
|
||||
if not isinstance(config, PretrainedConfig):
|
||||
config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
|
||||
|
||||
for config_class, model_class in TF_MODEL_FOR_PRETRAINING_MAPPING.items():
|
||||
if isinstance(config, config_class):
|
||||
return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
|
||||
raise ValueError(
|
||||
"Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
|
||||
"Model type should be one of {}.".format(
|
||||
config.__class__, cls.__name__, ", ".join(c.__name__ for c in TF_MODEL_FOR_PRETRAINING_MAPPING.keys())
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class TFAutoModelWithLMHead(object):
|
||||
r"""
|
||||
:class:`~transformers.TFAutoModelWithLMHead` is a generic model class
|
||||
@@ -548,7 +383,7 @@ class TFAutoModelWithLMHead(object):
|
||||
Examples::
|
||||
|
||||
config = BertConfig.from_pretrained('bert-base-uncased') # Download configuration from S3 and cache.
|
||||
model = TFAutoModelWithLMHead.from_config(config) # E.g. model was saved using `save_pretrained('./test/saved_model/')`
|
||||
model = AutoModelWithLMHead.from_config(config) # E.g. model was saved using `save_pretrained('./test/saved_model/')`
|
||||
"""
|
||||
for config_class, model_class in TF_MODEL_WITH_LM_HEAD_MAPPING.items():
|
||||
if isinstance(config, config_class):
|
||||
|
||||
@@ -49,7 +49,6 @@ TF_BERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
|
||||
"bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-tf_model.h5",
|
||||
"bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/tf_model.h5",
|
||||
"bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/tf_model.h5",
|
||||
"bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/tf_model.h5",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -1,118 +0,0 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
|
||||
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
""" TF 2.0 RoBERTa model. """
|
||||
|
||||
|
||||
import logging
|
||||
|
||||
from .configuration_camembert import CamembertConfig
|
||||
from .file_utils import add_start_docstrings
|
||||
from .modeling_tf_roberta import (
|
||||
TFRobertaForMaskedLM,
|
||||
TFRobertaForSequenceClassification,
|
||||
TFRobertaForTokenClassification,
|
||||
TFRobertaModel,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP = {}
|
||||
|
||||
|
||||
CAMEMBERT_START_DOCSTRING = r"""
|
||||
|
||||
.. note::
|
||||
|
||||
TF 2.0 models accepts two formats as inputs:
|
||||
|
||||
- having all inputs as keyword arguments (like PyTorch models), or
|
||||
- having all inputs as a list, tuple or dict in the first positional arguments.
|
||||
|
||||
This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having
|
||||
all the tensors in the first argument of the model call function: :obj:`model(inputs)`.
|
||||
|
||||
If you choose this second option, there are three possibilities you can use to gather all the input Tensors
|
||||
in the first positional argument :
|
||||
|
||||
- a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)`
|
||||
- a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
|
||||
:obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])`
|
||||
- a dictionary with one or several input Tensors associated to the input names given in the docstring:
|
||||
:obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})`
|
||||
|
||||
Parameters:
|
||||
config (:class:`~transformers.CamembertConfig`): Model configuration class with all the parameters of the
|
||||
model. Initializing with a config file does not load the weights associated with the model, only the configuration.
|
||||
Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights.
|
||||
"""
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"The bare CamemBERT Model transformer outputting raw hidden-states without any specific head on top.",
|
||||
CAMEMBERT_START_DOCSTRING,
|
||||
)
|
||||
class TFCamembertModel(TFRobertaModel):
|
||||
"""
|
||||
This class overrides :class:`~transformers.TFRobertaModel`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = CamembertConfig
|
||||
pretrained_model_archive_map = TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"""CamemBERT Model with a `language modeling` head on top. """, CAMEMBERT_START_DOCSTRING,
|
||||
)
|
||||
class TFCamembertForMaskedLM(TFRobertaForMaskedLM):
|
||||
"""
|
||||
This class overrides :class:`~transformers.TFRobertaForMaskedLM`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = CamembertConfig
|
||||
pretrained_model_archive_map = TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"""CamemBERT Model transformer with a sequence classification/regression head on top (a linear layer
|
||||
on top of the pooled output) e.g. for GLUE tasks. """,
|
||||
CAMEMBERT_START_DOCSTRING,
|
||||
)
|
||||
class TFCamembertForSequenceClassification(TFRobertaForSequenceClassification):
|
||||
"""
|
||||
This class overrides :class:`~transformers.TFRobertaForSequenceClassification`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = CamembertConfig
|
||||
pretrained_model_archive_map = TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"""CamemBERT Model with a token classification head on top (a linear layer on top of
|
||||
the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. """,
|
||||
CAMEMBERT_START_DOCSTRING,
|
||||
)
|
||||
class TFCamembertForTokenClassification(TFRobertaForTokenClassification):
|
||||
"""
|
||||
This class overrides :class:`~transformers.TFRobertaForTokenClassification`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = CamembertConfig
|
||||
pretrained_model_archive_map = TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
@@ -1,118 +0,0 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2019 Facebook AI Research and the HuggingFace Inc. team.
|
||||
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
""" TF 2.0 XLM-RoBERTa model. """
|
||||
|
||||
|
||||
import logging
|
||||
|
||||
from .configuration_xlm_roberta import XLMRobertaConfig
|
||||
from .file_utils import add_start_docstrings
|
||||
from .modeling_tf_roberta import (
|
||||
TFRobertaForMaskedLM,
|
||||
TFRobertaForSequenceClassification,
|
||||
TFRobertaForTokenClassification,
|
||||
TFRobertaModel,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP = {}
|
||||
|
||||
|
||||
XLM_ROBERTA_START_DOCSTRING = r"""
|
||||
|
||||
.. note::
|
||||
|
||||
TF 2.0 models accepts two formats as inputs:
|
||||
|
||||
- having all inputs as keyword arguments (like PyTorch models), or
|
||||
- having all inputs as a list, tuple or dict in the first positional arguments.
|
||||
|
||||
This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having
|
||||
all the tensors in the first argument of the model call function: :obj:`model(inputs)`.
|
||||
|
||||
If you choose this second option, there are three possibilities you can use to gather all the input Tensors
|
||||
in the first positional argument :
|
||||
|
||||
- a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)`
|
||||
- a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
|
||||
:obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])`
|
||||
- a dictionary with one or several input Tensors associated to the input names given in the docstring:
|
||||
:obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})`
|
||||
|
||||
Parameters:
|
||||
config (:class:`~transformers.XLMRobertaConfig`): Model configuration class with all the parameters of the
|
||||
model. Initializing with a config file does not load the weights associated with the model, only the configuration.
|
||||
Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights.
|
||||
"""
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"The bare XLM-RoBERTa Model transformer outputting raw hidden-states without any specific head on top.",
|
||||
XLM_ROBERTA_START_DOCSTRING,
|
||||
)
|
||||
class TFXLMRobertaModel(TFRobertaModel):
|
||||
"""
|
||||
This class overrides :class:`~transformers.TFRobertaModel`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = XLMRobertaConfig
|
||||
pretrained_model_archive_map = TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"""XLM-RoBERTa Model with a `language modeling` head on top. """, XLM_ROBERTA_START_DOCSTRING,
|
||||
)
|
||||
class TFXLMRobertaForMaskedLM(TFRobertaForMaskedLM):
|
||||
"""
|
||||
This class overrides :class:`~transformers.TFRobertaForMaskedLM`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = XLMRobertaConfig
|
||||
pretrained_model_archive_map = TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"""XLM-RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer
|
||||
on top of the pooled output) e.g. for GLUE tasks. """,
|
||||
XLM_ROBERTA_START_DOCSTRING,
|
||||
)
|
||||
class TFXLMRobertaForSequenceClassification(TFRobertaForSequenceClassification):
|
||||
"""
|
||||
This class overrides :class:`~transformers.TFRobertaForSequenceClassification`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = XLMRobertaConfig
|
||||
pretrained_model_archive_map = TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
|
||||
@add_start_docstrings(
|
||||
"""XLM-RoBERTa Model with a token classification head on top (a linear layer on top of
|
||||
the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. """,
|
||||
XLM_ROBERTA_START_DOCSTRING,
|
||||
)
|
||||
class TFXLMRobertaForTokenClassification(TFRobertaForTokenClassification):
|
||||
"""
|
||||
This class overrides :class:`~transformers.TFRobertaForTokenClassification`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
config_class = XLMRobertaConfig
|
||||
pretrained_model_archive_map = TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
@@ -284,9 +284,6 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin):
|
||||
# Only save the model itself if we are using distributed training
|
||||
model_to_save = self.module if hasattr(self, "module") else self
|
||||
|
||||
# Attach architecture to the config
|
||||
model_to_save.config.architectures = [model_to_save.__class__.__name__]
|
||||
|
||||
# Save configuration file
|
||||
model_to_save.config.save_pretrained(save_directory)
|
||||
|
||||
@@ -586,7 +583,7 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin):
|
||||
self,
|
||||
input_ids=None,
|
||||
max_length=None,
|
||||
do_sample=True,
|
||||
do_sample=None,
|
||||
num_beams=None,
|
||||
temperature=None,
|
||||
top_k=None,
|
||||
@@ -617,7 +614,7 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin):
|
||||
The max length of the sequence to be generated. Between 1 and infinity. Default to 20.
|
||||
|
||||
do_sample: (`optional`) bool
|
||||
If set to `False` greedy decoding is used. Otherwise sampling is used. Defaults to `True`.
|
||||
If set to `False` greedy decoding is used. Otherwise sampling is used. Default to greedy sampling.
|
||||
|
||||
num_beams: (`optional`) int
|
||||
Number of beams for beam search. Must be between 1 and infinity. 1 means no beam search. Default to 1.
|
||||
|
||||
@@ -756,7 +756,7 @@ class XLNetModel(XLNetPreTrainedModel):
|
||||
input_ids = input_ids.transpose(0, 1).contiguous()
|
||||
qlen, bsz = input_ids.shape[0], input_ids.shape[1]
|
||||
elif inputs_embeds is not None:
|
||||
inputs_embeds = inputs_embeds.transpose(0, 1).contiguous()
|
||||
inputs_embeds.transpose(0, 1).contiguous()
|
||||
qlen, bsz = inputs_embeds.shape[0], inputs_embeds.shape[1]
|
||||
else:
|
||||
raise ValueError("You have to specify either input_ids or inputs_embeds")
|
||||
|
||||
+13
-105
@@ -28,10 +28,7 @@ from typing import Dict, List, Optional, Tuple, Union
|
||||
import numpy as np
|
||||
|
||||
from .configuration_auto import ALL_PRETRAINED_CONFIG_ARCHIVE_MAP, AutoConfig
|
||||
from .configuration_distilbert import DistilBertConfig
|
||||
from .configuration_roberta import RobertaConfig
|
||||
from .configuration_utils import PretrainedConfig
|
||||
from .configuration_xlm import XLMConfig
|
||||
from .data import SquadExample, squad_convert_examples_to_features
|
||||
from .file_utils import is_tf_available, is_torch_available
|
||||
from .modelcard import ModelCard
|
||||
@@ -47,7 +44,6 @@ if is_tf_available():
|
||||
TFAutoModelForSequenceClassification,
|
||||
TFAutoModelForQuestionAnswering,
|
||||
TFAutoModelForTokenClassification,
|
||||
TFAutoModelWithLMHead,
|
||||
)
|
||||
|
||||
if is_torch_available():
|
||||
@@ -57,7 +53,6 @@ if is_torch_available():
|
||||
AutoModelForSequenceClassification,
|
||||
AutoModelForQuestionAnswering,
|
||||
AutoModelForTokenClassification,
|
||||
AutoModelWithLMHead,
|
||||
)
|
||||
|
||||
|
||||
@@ -69,7 +64,7 @@ def get_framework(model=None):
|
||||
If both frameworks are installed and no specific model is provided, defaults to using PyTorch.
|
||||
"""
|
||||
if is_tf_available() and is_torch_available() and model is not None and not isinstance(model, str):
|
||||
# Both framework are available but the user supplied a model class instance.
|
||||
# Both framework are available but the use supplied a model class instance.
|
||||
# Try to guess which framework to use from the model classname
|
||||
framework = "tf" if model.__class__.__name__.startswith("TF") else "pt"
|
||||
elif not is_tf_available() and not is_torch_available():
|
||||
@@ -326,7 +321,7 @@ class Pipeline(_ScikitCompat):
|
||||
self,
|
||||
model,
|
||||
tokenizer: PreTrainedTokenizer = None,
|
||||
modelcard: Optional[ModelCard] = None,
|
||||
modelcard: ModelCard = None,
|
||||
framework: Optional[str] = None,
|
||||
args_parser: ArgumentHandler = None,
|
||||
device: int = -1,
|
||||
@@ -358,8 +353,7 @@ class Pipeline(_ScikitCompat):
|
||||
|
||||
self.model.save_pretrained(save_directory)
|
||||
self.tokenizer.save_pretrained(save_directory)
|
||||
if self.modelcard is not None:
|
||||
self.modelcard.save_pretrained(save_directory)
|
||||
self.modelcard.save_pretrained(save_directory)
|
||||
|
||||
def transform(self, X):
|
||||
"""
|
||||
@@ -370,6 +364,7 @@ class Pipeline(_ScikitCompat):
|
||||
def predict(self, X):
|
||||
"""
|
||||
Scikit / Keras interface to transformers' pipelines. This method will forward to __call__().
|
||||
Se
|
||||
"""
|
||||
return self(X=X)
|
||||
|
||||
@@ -411,8 +406,9 @@ class Pipeline(_ScikitCompat):
|
||||
dict holding all the required parameters for model's forward
|
||||
"""
|
||||
args = ["input_ids", "attention_mask"]
|
||||
model_type = type(self.model).__name__.lower()
|
||||
|
||||
if not isinstance(self.model.config, (DistilBertConfig, XLMConfig, RobertaConfig)):
|
||||
if "distilbert" not in model_type and "xlm" not in model_type:
|
||||
args += ["token_type_ids"]
|
||||
|
||||
# PR #1548 (CLI) There is an issue with attention_mask
|
||||
@@ -424,10 +420,7 @@ class Pipeline(_ScikitCompat):
|
||||
else:
|
||||
return {k: [feature[k] for feature in features] for k in args}
|
||||
|
||||
def _parse_and_tokenize(self, *texts, **kwargs):
|
||||
"""
|
||||
Parse arguments and tokenize
|
||||
"""
|
||||
def __call__(self, *texts, **kwargs):
|
||||
# Parse arguments
|
||||
inputs = self._args_parser(*texts, **kwargs)
|
||||
inputs = self.tokenizer.batch_encode_plus(
|
||||
@@ -436,19 +429,13 @@ class Pipeline(_ScikitCompat):
|
||||
|
||||
# Filter out features not available on specific models
|
||||
inputs = self.inputs_for_model(inputs)
|
||||
|
||||
return inputs
|
||||
|
||||
def __call__(self, *texts, **kwargs):
|
||||
inputs = self._parse_and_tokenize(*texts, **kwargs)
|
||||
return self._forward(inputs)
|
||||
|
||||
def _forward(self, inputs, return_tensors=False):
|
||||
def _forward(self, inputs):
|
||||
"""
|
||||
Internal framework specific forward dispatching.
|
||||
Args:
|
||||
inputs: dict holding all the keyworded arguments for required by the model forward method.
|
||||
return_tensors: Whether to return native framework (pt/tf) tensors rather than numpy array.
|
||||
Returns:
|
||||
Numpy array
|
||||
"""
|
||||
@@ -462,10 +449,7 @@ class Pipeline(_ScikitCompat):
|
||||
inputs = self.ensure_tensor_on_device(**inputs)
|
||||
predictions = self.model(**inputs)[0].cpu()
|
||||
|
||||
if return_tensors:
|
||||
return predictions
|
||||
else:
|
||||
return predictions.numpy()
|
||||
return predictions.numpy()
|
||||
|
||||
|
||||
class FeatureExtractionPipeline(Pipeline):
|
||||
@@ -477,7 +461,7 @@ class FeatureExtractionPipeline(Pipeline):
|
||||
self,
|
||||
model,
|
||||
tokenizer: PreTrainedTokenizer = None,
|
||||
modelcard: Optional[ModelCard] = None,
|
||||
modelcard: ModelCard = None,
|
||||
framework: Optional[str] = None,
|
||||
args_parser: ArgumentHandler = None,
|
||||
device: int = -1,
|
||||
@@ -507,71 +491,6 @@ class TextClassificationPipeline(Pipeline):
|
||||
return [{"label": self.model.config.id2label[item.argmax()], "score": item.max()} for item in scores]
|
||||
|
||||
|
||||
class FillMaskPipeline(Pipeline):
|
||||
"""
|
||||
Masked language modeling prediction pipeline using ModelWithLMHead head.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model,
|
||||
tokenizer: PreTrainedTokenizer = None,
|
||||
modelcard: Optional[ModelCard] = None,
|
||||
framework: Optional[str] = None,
|
||||
args_parser: ArgumentHandler = None,
|
||||
device: int = -1,
|
||||
topk=5,
|
||||
):
|
||||
super().__init__(
|
||||
model=model,
|
||||
tokenizer=tokenizer,
|
||||
modelcard=modelcard,
|
||||
framework=framework,
|
||||
args_parser=args_parser,
|
||||
device=device,
|
||||
binary_output=True,
|
||||
)
|
||||
|
||||
self.topk = topk
|
||||
|
||||
def __call__(self, *args, **kwargs):
|
||||
inputs = self._parse_and_tokenize(*args, **kwargs)
|
||||
outputs = self._forward(inputs, return_tensors=True)
|
||||
|
||||
results = []
|
||||
batch_size = outputs.shape[0] if self.framework == "tf" else outputs.size(0)
|
||||
|
||||
for i in range(batch_size):
|
||||
input_ids = inputs["input_ids"][i]
|
||||
result = []
|
||||
|
||||
if self.framework == "tf":
|
||||
masked_index = tf.where(input_ids == self.tokenizer.mask_token_id).numpy().item()
|
||||
logits = outputs[i, masked_index, :]
|
||||
probs = tf.nn.softmax(logits)
|
||||
topk = tf.math.top_k(probs, k=self.topk)
|
||||
values, predictions = topk.values.numpy(), topk.indices.numpy()
|
||||
else:
|
||||
masked_index = (input_ids == self.tokenizer.mask_token_id).nonzero().item()
|
||||
logits = outputs[i, masked_index, :]
|
||||
probs = logits.softmax(dim=0)
|
||||
values, predictions = probs.topk(self.topk)
|
||||
|
||||
for v, p in zip(values.tolist(), predictions.tolist()):
|
||||
tokens = input_ids.numpy()
|
||||
tokens[masked_index] = p
|
||||
# Filter padding out:
|
||||
tokens = tokens[np.where(tokens != self.tokenizer.pad_token_id)]
|
||||
result.append({"sequence": self.tokenizer.decode(tokens), "score": v, "token": p})
|
||||
|
||||
# Append
|
||||
results += [result]
|
||||
|
||||
if len(results) == 1:
|
||||
return results[0]
|
||||
return results
|
||||
|
||||
|
||||
class NerPipeline(Pipeline):
|
||||
"""
|
||||
Named Entity Recognition pipeline using ModelForTokenClassification head.
|
||||
@@ -583,7 +502,7 @@ class NerPipeline(Pipeline):
|
||||
self,
|
||||
model,
|
||||
tokenizer: PreTrainedTokenizer = None,
|
||||
modelcard: Optional[ModelCard] = None,
|
||||
modelcard: ModelCard = None,
|
||||
framework: Optional[str] = None,
|
||||
args_parser: ArgumentHandler = None,
|
||||
device: int = -1,
|
||||
@@ -604,8 +523,7 @@ class NerPipeline(Pipeline):
|
||||
self.ignore_labels = ignore_labels
|
||||
|
||||
def __call__(self, *texts, **kwargs):
|
||||
inputs = self._args_parser(*texts, **kwargs)
|
||||
answers = []
|
||||
inputs, answers = self._args_parser(*texts, **kwargs), []
|
||||
for sentence in inputs:
|
||||
|
||||
# Manage correct placement of the tensors
|
||||
@@ -722,7 +640,7 @@ class QuestionAnsweringPipeline(Pipeline):
|
||||
self,
|
||||
model,
|
||||
tokenizer: Optional[PreTrainedTokenizer],
|
||||
modelcard: Optional[ModelCard] = None,
|
||||
modelcard: Optional[ModelCard],
|
||||
framework: Optional[str] = None,
|
||||
device: int = -1,
|
||||
**kwargs
|
||||
@@ -985,16 +903,6 @@ SUPPORTED_TASKS = {
|
||||
"tokenizer": "distilbert-base-uncased",
|
||||
},
|
||||
},
|
||||
"fill-mask": {
|
||||
"impl": FillMaskPipeline,
|
||||
"tf": TFAutoModelWithLMHead if is_tf_available() else None,
|
||||
"pt": AutoModelWithLMHead if is_torch_available() else None,
|
||||
"default": {
|
||||
"model": {"pt": "distilroberta-base", "tf": "distilroberta-base"},
|
||||
"config": None,
|
||||
"tokenizer": "distilroberta-base",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -25,7 +25,6 @@ from .configuration_auto import (
|
||||
CamembertConfig,
|
||||
CTRLConfig,
|
||||
DistilBertConfig,
|
||||
FlaubertConfig,
|
||||
GPT2Config,
|
||||
OpenAIGPTConfig,
|
||||
RobertaConfig,
|
||||
@@ -37,15 +36,14 @@ from .configuration_auto import (
|
||||
)
|
||||
from .configuration_utils import PretrainedConfig
|
||||
from .tokenization_albert import AlbertTokenizer
|
||||
from .tokenization_bert import BertTokenizer
|
||||
from .tokenization_bert import BertTokenizer, BertTokenizerFast
|
||||
from .tokenization_bert_japanese import BertJapaneseTokenizer
|
||||
from .tokenization_camembert import CamembertTokenizer
|
||||
from .tokenization_ctrl import CTRLTokenizer
|
||||
from .tokenization_distilbert import DistilBertTokenizer
|
||||
from .tokenization_flaubert import FlaubertTokenizer
|
||||
from .tokenization_gpt2 import GPT2Tokenizer
|
||||
from .tokenization_openai import OpenAIGPTTokenizer
|
||||
from .tokenization_roberta import RobertaTokenizer
|
||||
from .tokenization_ctrl import CTRLTokenizer, CTRLTokenizerFast
|
||||
from .tokenization_distilbert import DistilBertTokenizer, DistilBertTokenizerFast
|
||||
from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast
|
||||
from .tokenization_openai import OpenAIGPTTokenizer, OpenAIGPTTokenizerFast
|
||||
from .tokenization_roberta import RobertaTokenizer, RobertaTokenizerFast
|
||||
from .tokenization_t5 import T5Tokenizer
|
||||
from .tokenization_transfo_xl import TransfoXLTokenizer
|
||||
from .tokenization_xlm import XLMTokenizer
|
||||
@@ -58,25 +56,24 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
TOKENIZER_MAPPING = OrderedDict(
|
||||
[
|
||||
(T5Config, T5Tokenizer),
|
||||
(DistilBertConfig, DistilBertTokenizer),
|
||||
(AlbertConfig, AlbertTokenizer),
|
||||
(CamembertConfig, CamembertTokenizer),
|
||||
(XLMRobertaConfig, XLMRobertaTokenizer),
|
||||
(RobertaConfig, RobertaTokenizer),
|
||||
(BertConfig, BertTokenizer),
|
||||
(OpenAIGPTConfig, OpenAIGPTTokenizer),
|
||||
(GPT2Config, GPT2Tokenizer),
|
||||
(TransfoXLConfig, TransfoXLTokenizer),
|
||||
(XLNetConfig, XLNetTokenizer),
|
||||
(FlaubertConfig, FlaubertTokenizer),
|
||||
(XLMConfig, XLMTokenizer),
|
||||
(CTRLConfig, CTRLTokenizer),
|
||||
(T5Config, (T5Tokenizer, None)),
|
||||
(DistilBertConfig, (DistilBertTokenizer, DistilBertTokenizerFast)),
|
||||
(AlbertConfig, (AlbertTokenizer, None)),
|
||||
(CamembertConfig, (CamembertTokenizer, None)),
|
||||
(XLMRobertaConfig, (XLMRobertaTokenizer, None)),
|
||||
(RobertaConfig, (RobertaTokenizer, RobertaTokenizerFast)),
|
||||
(BertConfig, (BertTokenizer, BertTokenizerFast)),
|
||||
(OpenAIGPTConfig, (OpenAIGPTTokenizer, OpenAIGPTTokenizerFast)),
|
||||
(GPT2Config, (GPT2Tokenizer, GPT2TokenizerFast)),
|
||||
(TransfoXLConfig, (TransfoXLTokenizer, None)),
|
||||
(XLNetConfig, (XLNetTokenizer, None)),
|
||||
(XLMConfig, (XLMTokenizer, None)),
|
||||
(CTRLConfig, (CTRLTokenizer, CTRLTokenizerFast)),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
class AutoTokenizer:
|
||||
class AutoTokenizer(object):
|
||||
r""":class:`~transformers.AutoTokenizer` is a generic tokenizer class
|
||||
that will be instantiated as one of the tokenizer classes of the library
|
||||
when created with the `AutoTokenizer.from_pretrained(pretrained_model_name_or_path)`
|
||||
@@ -177,9 +174,12 @@ class AutoTokenizer:
|
||||
if "bert-base-japanese" in pretrained_model_name_or_path:
|
||||
return BertJapaneseTokenizer.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
|
||||
|
||||
for config_class, tokenizer_class in TOKENIZER_MAPPING.items():
|
||||
for config_class, (tokenizer_class_py, tokenizer_class_ru) in TOKENIZER_MAPPING.items():
|
||||
if isinstance(config, config_class):
|
||||
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
|
||||
if tokenizer_class_ru:
|
||||
return tokenizer_class_ru.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
|
||||
else:
|
||||
return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
|
||||
|
||||
raise ValueError(
|
||||
"Unrecognized configuration class {} to build an AutoTokenizer.\n"
|
||||
|
||||
@@ -48,7 +48,6 @@ PRETRAINED_VOCAB_FILES_MAP = {
|
||||
"bert-base-german-dbmdz-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-vocab.txt",
|
||||
"bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/vocab.txt",
|
||||
"bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/vocab.txt",
|
||||
"bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/vocab.txt",
|
||||
}
|
||||
}
|
||||
|
||||
@@ -70,7 +69,6 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
|
||||
"bert-base-german-dbmdz-uncased": 512,
|
||||
"bert-base-finnish-cased-v1": 512,
|
||||
"bert-base-finnish-uncased-v1": 512,
|
||||
"bert-base-dutch-cased": 512,
|
||||
}
|
||||
|
||||
PRETRAINED_INIT_CONFIGURATION = {
|
||||
@@ -91,7 +89,6 @@ PRETRAINED_INIT_CONFIGURATION = {
|
||||
"bert-base-german-dbmdz-uncased": {"do_lower_case": True},
|
||||
"bert-base-finnish-cased-v1": {"do_lower_case": False},
|
||||
"bert-base-finnish-uncased-v1": {"do_lower_case": True},
|
||||
"bert-base-dutch-cased": {"do_lower_case": False},
|
||||
}
|
||||
|
||||
|
||||
@@ -558,6 +555,15 @@ class BertTokenizerFast(PreTrainedTokenizerFast):
|
||||
**kwargs
|
||||
):
|
||||
super().__init__(
|
||||
tk.implementations.BertWordPieceTokenizer(
|
||||
vocab_file,
|
||||
add_special_tokens,
|
||||
unk_token,
|
||||
sep_token,
|
||||
cls_token,
|
||||
handle_chinese_chars=tokenize_chinese_chars,
|
||||
lowercase=do_lower_case,
|
||||
),
|
||||
unk_token=unk_token,
|
||||
sep_token=sep_token,
|
||||
pad_token=pad_token,
|
||||
@@ -565,33 +571,3 @@ class BertTokenizerFast(PreTrainedTokenizerFast):
|
||||
mask_token=mask_token,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
self._tokenizer = tk.Tokenizer(tk.models.WordPiece.from_files(vocab_file, unk_token=unk_token))
|
||||
self._update_special_tokens()
|
||||
self._tokenizer.with_pre_tokenizer(
|
||||
tk.pre_tokenizers.BertPreTokenizer.new(
|
||||
do_basic_tokenize=do_basic_tokenize,
|
||||
do_lower_case=do_lower_case,
|
||||
tokenize_chinese_chars=tokenize_chinese_chars,
|
||||
never_split=never_split if never_split is not None else [],
|
||||
)
|
||||
)
|
||||
self._tokenizer.with_decoder(tk.decoders.WordPiece.new())
|
||||
|
||||
if add_special_tokens:
|
||||
self._tokenizer.with_post_processor(
|
||||
tk.processors.BertProcessing.new(
|
||||
(sep_token, self._tokenizer.token_to_id(sep_token)),
|
||||
(cls_token, self._tokenizer.token_to_id(cls_token)),
|
||||
)
|
||||
)
|
||||
if max_length is not None:
|
||||
self._tokenizer.with_truncation(max_length, stride=stride, strategy=truncation_strategy)
|
||||
self._tokenizer.with_padding(
|
||||
max_length=max_length if pad_to_max_length else None,
|
||||
direction=self.padding_side,
|
||||
pad_id=self.pad_token_id,
|
||||
pad_type_id=self.pad_token_type_id,
|
||||
pad_token=self.pad_token,
|
||||
)
|
||||
self._decoder = tk.decoders.WordPiece.new()
|
||||
|
||||
@@ -40,13 +40,6 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
|
||||
"camembert-base": None,
|
||||
}
|
||||
|
||||
SHARED_MODEL_IDENTIFIERS = [
|
||||
# Load with
|
||||
# `tokenizer = AutoTokenizer.from_pretrained("username/pretrained_model")`
|
||||
"Musixmatch/umberto-commoncrawl-cased-v1",
|
||||
"Musixmatch/umberto-wikipedia-uncased-v1",
|
||||
]
|
||||
|
||||
|
||||
class CamembertTokenizer(PreTrainedTokenizer):
|
||||
"""
|
||||
|
||||
@@ -20,8 +20,9 @@ import logging
|
||||
import os
|
||||
|
||||
import regex as re
|
||||
from tokenizers import BPETokenizer
|
||||
|
||||
from .tokenization_utils import PreTrainedTokenizer
|
||||
from .tokenization_utils import PreTrainedTokenizer, PreTrainedTokenizerFast
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -32,8 +33,8 @@ VOCAB_FILES_NAMES = {
|
||||
}
|
||||
|
||||
PRETRAINED_VOCAB_FILES_MAP = {
|
||||
"vocab_file": {"ctrl": "https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-vocab.json"},
|
||||
"merges_file": {"ctrl": "https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-merges.txt"},
|
||||
"vocab_file": {"ctrl": "https://s3.amazonaws.com/models.huggingface.co/bert/ctrl-vocab.json"},
|
||||
"merges_file": {"ctrl": "https://s3.amazonaws.com/models.huggingface.co/bert/ctrl-merges.txt"},
|
||||
}
|
||||
|
||||
PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
|
||||
@@ -148,14 +149,14 @@ class CTRLTokenizer(PreTrainedTokenizer):
|
||||
return len(self.encoder)
|
||||
|
||||
def bpe(self, token):
|
||||
if token in self.cache:
|
||||
return self.cache[token]
|
||||
word = tuple(token)
|
||||
word = tuple(list(word[:-1]) + [word[-1] + "</w>"])
|
||||
if token in self.cache:
|
||||
return self.cache[token]
|
||||
pairs = get_pairs(word)
|
||||
|
||||
if not pairs:
|
||||
return token
|
||||
return token + "</w>"
|
||||
|
||||
while True:
|
||||
bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf")))
|
||||
@@ -186,8 +187,9 @@ class CTRLTokenizer(PreTrainedTokenizer):
|
||||
break
|
||||
else:
|
||||
pairs = get_pairs(word)
|
||||
word = "@@ ".join(word)
|
||||
word = word[:-4]
|
||||
word = " ".join(word)
|
||||
if word == "\n </w>":
|
||||
word = "\n</w>"
|
||||
self.cache[token] = word
|
||||
return word
|
||||
|
||||
@@ -212,7 +214,7 @@ class CTRLTokenizer(PreTrainedTokenizer):
|
||||
|
||||
def convert_tokens_to_string(self, tokens):
|
||||
""" Converts a sequence of tokens (string) in a single string. """
|
||||
out_string = " ".join(tokens).replace("@@ ", "").strip()
|
||||
out_string = "".join(tokens).replace("</w>", " ").strip()
|
||||
return out_string
|
||||
|
||||
def save_vocabulary(self, save_directory):
|
||||
@@ -246,3 +248,13 @@ class CTRLTokenizer(PreTrainedTokenizer):
|
||||
# tokens_generated_so_far = re.sub('(@@ )', '', string=filtered_tokens)
|
||||
# tokens_generated_so_far = re.sub('(@@ ?$)', '', string=tokens_generated_so_far)
|
||||
# return ''.join(tokens_generated_so_far)
|
||||
|
||||
|
||||
class CTRLTokenizerFast(PreTrainedTokenizerFast):
|
||||
vocab_files_names = VOCAB_FILES_NAMES
|
||||
pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
|
||||
max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
|
||||
control_codes = CONTROL_CODES
|
||||
|
||||
def __init__(self, vocab_file, merges_file, unk_token="<unk>", **kwargs):
|
||||
super().__init__(BPETokenizer(vocab_file, merges_file, unk_token), **kwargs)
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
|
||||
import logging
|
||||
|
||||
from .tokenization_bert import BertTokenizer
|
||||
from .tokenization_bert import BertTokenizer, BertTokenizerFast
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -68,3 +68,10 @@ class DistilBertTokenizer(BertTokenizer):
|
||||
pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
|
||||
max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
|
||||
pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION
|
||||
|
||||
|
||||
class DistilBertTokenizerFast(BertTokenizerFast):
|
||||
vocab_files_names = VOCAB_FILES_NAMES
|
||||
pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
|
||||
max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
|
||||
pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION
|
||||
|
||||
@@ -1,145 +0,0 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2019-present CNRS, Facebook Inc. and the HuggingFace Inc. team.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
"""Tokenization classes for Flaubert, based on XLM."""
|
||||
|
||||
|
||||
import logging
|
||||
import unicodedata
|
||||
|
||||
import six
|
||||
|
||||
from .tokenization_xlm import XLMTokenizer
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
VOCAB_FILES_NAMES = {
|
||||
"vocab_file": "vocab.json",
|
||||
"merges_file": "merges.txt",
|
||||
}
|
||||
|
||||
PRETRAINED_VOCAB_FILES_MAP = {
|
||||
"vocab_file": {
|
||||
"flaubert-small-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_small_cased/vocab.json",
|
||||
"flaubert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/vocab.json",
|
||||
"flaubert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_cased/vocab.json",
|
||||
"flaubert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_large_cased/vocab.json",
|
||||
},
|
||||
"merges_file": {
|
||||
"flaubert-small-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_small_cased/merges.txt",
|
||||
"flaubert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/merges.txt",
|
||||
"flaubert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_cased/merges.txt",
|
||||
"flaubert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_large_cased/merges.txt",
|
||||
},
|
||||
}
|
||||
|
||||
PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
|
||||
"flaubert-small-cased": 512,
|
||||
"flaubert-base-uncased": 512,
|
||||
"flaubert-base-cased": 512,
|
||||
"flaubert-large-cased": 512,
|
||||
}
|
||||
|
||||
PRETRAINED_INIT_CONFIGURATION = {
|
||||
"flaubert-small-cased": {"do_lowercase": False},
|
||||
"flaubert-base-uncased": {"do_lowercase": True},
|
||||
"flaubert-base-cased": {"do_lowercase": False},
|
||||
"flaubert-large-cased": {"do_lowercase": False},
|
||||
}
|
||||
|
||||
|
||||
def convert_to_unicode(text):
|
||||
"""
|
||||
Converts `text` to Unicode (if it's not already), assuming UTF-8 input.
|
||||
"""
|
||||
# six_ensure_text is copied from https://github.com/benjaminp/six
|
||||
def six_ensure_text(s, encoding="utf-8", errors="strict"):
|
||||
if isinstance(s, six.binary_type):
|
||||
return s.decode(encoding, errors)
|
||||
elif isinstance(s, six.text_type):
|
||||
return s
|
||||
else:
|
||||
raise TypeError("not expecting type '%s'" % type(s))
|
||||
|
||||
return six_ensure_text(text, encoding="utf-8", errors="ignore")
|
||||
|
||||
|
||||
class FlaubertTokenizer(XLMTokenizer):
|
||||
"""
|
||||
BPE tokenizer for Flaubert
|
||||
|
||||
- Moses preprocessing & tokenization
|
||||
|
||||
- Normalize all inputs text
|
||||
|
||||
- argument ``special_tokens`` and function ``set_special_tokens``, can be used to add additional symbols \
|
||||
(ex: "__classify__") to a vocabulary
|
||||
|
||||
- `do_lowercase` controle lower casing (automatically set for pretrained vocabularies)
|
||||
"""
|
||||
|
||||
vocab_files_names = VOCAB_FILES_NAMES
|
||||
pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
|
||||
pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION
|
||||
max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
|
||||
|
||||
def __init__(self, do_lowercase=False, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
self.do_lowercase = do_lowercase
|
||||
self.do_lowercase_and_remove_accent = False
|
||||
|
||||
def preprocess_text(self, text):
|
||||
text = text.replace("``", '"').replace("''", '"')
|
||||
text = convert_to_unicode(text)
|
||||
text = unicodedata.normalize("NFC", text)
|
||||
|
||||
if self.do_lowercase:
|
||||
text = text.lower()
|
||||
|
||||
return text
|
||||
|
||||
def _tokenize(self, text, bypass_tokenizer=False):
|
||||
"""
|
||||
Tokenize a string given language code using Moses.
|
||||
|
||||
Details of tokenization:
|
||||
- [sacremoses](https://github.com/alvations/sacremoses): port of Moses
|
||||
- Install with `pip install sacremoses`
|
||||
|
||||
Args:
|
||||
- bypass_tokenizer: Allow users to preprocess and tokenize the sentences externally (default = False) (bool). If True, we only apply BPE.
|
||||
|
||||
Returns:
|
||||
List of tokens.
|
||||
"""
|
||||
lang = "fr"
|
||||
if lang and self.lang2id and lang not in self.lang2id:
|
||||
logger.error(
|
||||
"Supplied language code not found in lang2id mapping. Please check that your language is supported by the loaded pretrained model."
|
||||
)
|
||||
|
||||
if bypass_tokenizer:
|
||||
text = text.split()
|
||||
else:
|
||||
text = self.preprocess_text(text)
|
||||
text = self.moses_pipeline(text, lang=lang)
|
||||
text = self.moses_tokenize(text, lang=lang)
|
||||
|
||||
split_tokens = []
|
||||
for token in text:
|
||||
if token:
|
||||
split_tokens.extend([t for t in self.bpe(token).split(" ")])
|
||||
|
||||
return split_tokens
|
||||
@@ -22,6 +22,7 @@ from functools import lru_cache
|
||||
|
||||
import regex as re
|
||||
import tokenizers as tk
|
||||
from tokenizers import ByteLevelBPETokenizer
|
||||
|
||||
from .tokenization_utils import PreTrainedTokenizer, PreTrainedTokenizerFast
|
||||
|
||||
@@ -268,19 +269,25 @@ class GPT2TokenizerFast(PreTrainedTokenizerFast):
|
||||
truncation_strategy="longest_first",
|
||||
**kwargs
|
||||
):
|
||||
super().__init__(bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, **kwargs)
|
||||
|
||||
self._tokenizer = tk.Tokenizer(tk.models.BPE.from_files(vocab_file, merges_file))
|
||||
self._update_special_tokens()
|
||||
self._tokenizer.with_pre_tokenizer(tk.pre_tokenizers.ByteLevel.new(add_prefix_space=add_prefix_space))
|
||||
self._tokenizer.with_decoder(tk.decoders.ByteLevel.new())
|
||||
if max_length:
|
||||
self._tokenizer.with_truncation(max_length, stride=stride, strategy=truncation_strategy)
|
||||
self._tokenizer.with_padding(
|
||||
max_length=max_length if pad_to_max_length else None,
|
||||
direction=self.padding_side,
|
||||
pad_id=self.pad_token_id if self.pad_token_id is not None else 0,
|
||||
pad_type_id=self.pad_token_type_id,
|
||||
pad_token=self.pad_token if self.pad_token is not None else "",
|
||||
super().__init__(
|
||||
ByteLevelBPETokenizer(vocab_file, merges_file, add_prefix_space),
|
||||
bos_token=bos_token,
|
||||
eos_token=eos_token,
|
||||
unk_token=unk_token,
|
||||
**kwargs,
|
||||
)
|
||||
self._decoder = tk.decoders.ByteLevel.new()
|
||||
|
||||
# self._tokenizer = tk.Tokenizer(tk.models.BPE.from_files(vocab_file, merges_file))
|
||||
# self._update_special_tokens()
|
||||
# self._tokenizer.with_pre_tokenizer(tk.pre_tokenizers.ByteLevel.new(add_prefix_space=add_prefix_space))
|
||||
# self._tokenizer.with_decoder(tk.decoders.ByteLevel.new())
|
||||
# if max_length:
|
||||
# self._tokenizer.with_truncation(max_length, stride=stride, strategy=truncation_strategy)
|
||||
# self._tokenizer.with_padding(
|
||||
# max_length=max_length if pad_to_max_length else None,
|
||||
# direction=self.padding_side,
|
||||
# pad_id=self.pad_token_id if self.pad_token_id is not None else 0,
|
||||
# pad_type_id=self.pad_token_type_id,
|
||||
# pad_token=self.pad_token if self.pad_token is not None else "",
|
||||
# )
|
||||
# self._decoder = tk.decoders.ByteLevel.new()
|
||||
|
||||
@@ -20,8 +20,10 @@ import logging
|
||||
import os
|
||||
import re
|
||||
|
||||
from tokenizers import BPETokenizer
|
||||
|
||||
from .tokenization_bert import BasicTokenizer
|
||||
from .tokenization_utils import PreTrainedTokenizer
|
||||
from .tokenization_utils import PreTrainedTokenizer, PreTrainedTokenizerFast
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -213,3 +215,12 @@ class OpenAIGPTTokenizer(PreTrainedTokenizer):
|
||||
index += 1
|
||||
|
||||
return vocab_file, merge_file
|
||||
|
||||
|
||||
class OpenAIGPTTokenizerFast(PreTrainedTokenizerFast):
|
||||
vocab_files_names = VOCAB_FILES_NAMES
|
||||
pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
|
||||
max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
|
||||
|
||||
def __init__(self, vocab_file, merges_file, unk_token="<unk>", **kwargs):
|
||||
super().__init__(BPETokenizer(vocab_file, merges_file, unk_token), **kwargs)
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
|
||||
import logging
|
||||
|
||||
from .tokenization_gpt2 import GPT2Tokenizer
|
||||
from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -154,3 +154,30 @@ class RobertaTokenizer(GPT2Tokenizer):
|
||||
if token_ids_1 is None:
|
||||
return len(cls + token_ids_0 + sep) * [0]
|
||||
return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]
|
||||
|
||||
|
||||
class RobertaTokenizerFast(GPT2TokenizerFast):
|
||||
vocab_files_names = VOCAB_FILES_NAMES
|
||||
pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
|
||||
max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vocab_file,
|
||||
merges_file,
|
||||
errors="replace",
|
||||
bos_token="<s>",
|
||||
eos_token="</s>",
|
||||
sep_token="</s>",
|
||||
cls_token="<s>",
|
||||
unk_token="<unk>",
|
||||
pad_token="<pad>",
|
||||
mask_token="<mask>",
|
||||
**kwargs
|
||||
):
|
||||
kwargs["pad_token"] = pad_token
|
||||
kwargs["sep_token"] = sep_token
|
||||
kwargs["cls_token"] = cls_token
|
||||
kwargs["mask_token"] = mask_token
|
||||
|
||||
super().__init__(vocab_file, merges_file, unk_token, bos_token, eos_token, add_prefix_space=True)
|
||||
|
||||
@@ -21,6 +21,9 @@ import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from contextlib import contextmanager
|
||||
|
||||
from tokenizers.implementations import BaseTokenizer
|
||||
|
||||
from .file_utils import cached_path, hf_bucket_url, is_remote_url, is_tf_available, is_torch_available
|
||||
|
||||
@@ -37,6 +40,56 @@ ADDED_TOKENS_FILE = "added_tokens.json"
|
||||
TOKENIZER_CONFIG_FILE = "tokenizer_config.json"
|
||||
|
||||
|
||||
@contextmanager
|
||||
def truncate_and_pad(
|
||||
tokenizer: BaseTokenizer,
|
||||
max_length: int,
|
||||
stride: int,
|
||||
strategy: str,
|
||||
pad_to_max_length: bool,
|
||||
padding_side: str,
|
||||
pad_token_id: int,
|
||||
pad_token_type_id: int,
|
||||
pad_token: str,
|
||||
):
|
||||
"""
|
||||
This contextmanager is in charge of defining the truncation and the padding strategies and then
|
||||
restore the tokenizer settings afterwards.
|
||||
|
||||
:param tokenizer:
|
||||
:param max_length:
|
||||
:param stride:
|
||||
:param strategy:
|
||||
:param pad_to_max_length:
|
||||
:param padding_side:
|
||||
:param pad_token_id:
|
||||
:param pad_token_type_id:
|
||||
:param pad_token:
|
||||
:return:
|
||||
"""
|
||||
|
||||
# Handle all the truncation and padding stuff
|
||||
if max_length is not None:
|
||||
tokenizer.enable_truncation(max_length, stride=stride, strategy=strategy)
|
||||
|
||||
if pad_to_max_length:
|
||||
tokenizer.enable_padding(
|
||||
max_length=max_length,
|
||||
direction=padding_side,
|
||||
pad_id=pad_token_id,
|
||||
pad_type_id=pad_token_type_id,
|
||||
pad_token=pad_token,
|
||||
)
|
||||
|
||||
yield
|
||||
|
||||
if max_length is not None:
|
||||
tokenizer.no_truncation()
|
||||
|
||||
if pad_to_max_length:
|
||||
tokenizer.no_padding()
|
||||
|
||||
|
||||
class PreTrainedTokenizer(object):
|
||||
""" Base class for all tokenizers.
|
||||
Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary.
|
||||
@@ -326,7 +379,7 @@ class PreTrainedTokenizer(object):
|
||||
cls.pretrained_init_configuration
|
||||
and pretrained_model_name_or_path in cls.pretrained_init_configuration
|
||||
):
|
||||
init_configuration = cls.pretrained_init_configuration[pretrained_model_name_or_path].copy()
|
||||
init_configuration = cls.pretrained_init_configuration[pretrained_model_name_or_path]
|
||||
else:
|
||||
# Get the vocabulary from local files
|
||||
logger.info(
|
||||
@@ -832,6 +885,7 @@ class PreTrainedTokenizer(object):
|
||||
return_attention_mask=True,
|
||||
return_overflowing_tokens=False,
|
||||
return_special_tokens_mask=False,
|
||||
return_offsets_mapping=False,
|
||||
**kwargs
|
||||
):
|
||||
"""
|
||||
@@ -905,6 +959,9 @@ class PreTrainedTokenizer(object):
|
||||
"Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers."
|
||||
)
|
||||
|
||||
if return_offsets_mapping:
|
||||
logger.warning("offset mapping is not available on Python tokenizers.")
|
||||
|
||||
first_ids = get_input_ids(text)
|
||||
second_ids = get_input_ids(text_pair) if text_pair is not None else None
|
||||
|
||||
@@ -998,8 +1055,7 @@ class PreTrainedTokenizer(object):
|
||||
for key, value in batch_outputs.items():
|
||||
|
||||
padded_value = value
|
||||
# verify that the tokenizer has a pad_token_id
|
||||
if key != "input_len" and self._pad_token is not None:
|
||||
if key != "input_len":
|
||||
# Padding handle
|
||||
padded_value = [
|
||||
v + [self.pad_token_id if key == "input_ids" else 1] * (max_seq_len - len(v))
|
||||
@@ -1418,30 +1474,27 @@ class PreTrainedTokenizer(object):
|
||||
|
||||
|
||||
class PreTrainedTokenizerFast(PreTrainedTokenizer):
|
||||
_tokenizer = None
|
||||
_decoder = None
|
||||
def __init__(self, tokenizer: BaseTokenizer, **kwargs):
|
||||
if tokenizer is None:
|
||||
raise ValueError("Provided tokenizer cannot be None")
|
||||
self._tokenizer = tokenizer
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
|
||||
@property
|
||||
def tokenizer(self):
|
||||
if self._tokenizer is None:
|
||||
raise NotImplementedError
|
||||
return self._tokenizer
|
||||
|
||||
@property
|
||||
def decoder(self):
|
||||
if self._decoder is None:
|
||||
raise NotImplementedError
|
||||
return self._decoder
|
||||
return self._tokenizer._tokenizer.decoder
|
||||
|
||||
@property
|
||||
def vocab_size(self):
|
||||
return self.tokenizer.get_vocab_size(with_added_tokens=False)
|
||||
return self._tokenizer.get_vocab_size(with_added_tokens=False)
|
||||
|
||||
def __len__(self):
|
||||
return self.tokenizer.get_vocab_size(with_added_tokens=True)
|
||||
return self._tokenizer.get_vocab_size(with_added_tokens=True)
|
||||
|
||||
@PreTrainedTokenizer.bos_token.setter
|
||||
def bos_token(self, value):
|
||||
@@ -1495,36 +1548,59 @@ class PreTrainedTokenizerFast(PreTrainedTokenizer):
|
||||
return_attention_mask=True,
|
||||
return_overflowing_tokens=False,
|
||||
return_special_tokens_mask=False,
|
||||
return_offsets_mapping=False,
|
||||
pad_token_id: int = 0,
|
||||
pad_to_length: int = -1,
|
||||
):
|
||||
if return_overflowing_tokens and encoding.overflowing is not None:
|
||||
encodings = [encoding] + encoding.overflowing
|
||||
else:
|
||||
encodings = [encoding]
|
||||
|
||||
encoding_dict = {
|
||||
"input_ids": encoding.ids,
|
||||
"input_ids": [e.ids for e in encodings],
|
||||
}
|
||||
|
||||
if return_token_type_ids:
|
||||
encoding_dict["token_type_ids"] = encoding.type_ids
|
||||
encoding_dict["token_type_ids"] = [e.type_ids for e in encodings]
|
||||
if return_attention_mask:
|
||||
encoding_dict["attention_mask"] = encoding.attention_mask
|
||||
if return_overflowing_tokens:
|
||||
overflowing = encoding.overflowing
|
||||
encoding_dict["overflowing_tokens"] = overflowing.ids if overflowing is not None else []
|
||||
encoding_dict["attention_mask"] = [e.attention_mask for e in encodings]
|
||||
if return_special_tokens_mask:
|
||||
encoding_dict["special_tokens_mask"] = encoding.special_tokens_mask
|
||||
encoding_dict["special_tokens_mask"] = [e.special_tokens_mask for e in encodings]
|
||||
if return_offsets_mapping:
|
||||
encoding_dict["offset_mapping"] = [e.offsets for e in encodings]
|
||||
|
||||
if pad_to_length > 0:
|
||||
for i in range(len(encoding_dict["input_ids"])):
|
||||
if len(encoding_dict["input_ids"][i]) < pad_to_length:
|
||||
padding = pad_to_length - len(encoding_dict["input_ids"][i])
|
||||
encoding_dict["input_ids"][i] += [pad_token_id] * padding
|
||||
|
||||
if return_attention_mask:
|
||||
encoding_dict["attention_mask"][i] += [0] * padding
|
||||
|
||||
if return_special_tokens_mask:
|
||||
encoding_dict["special_tokens_mask"][i] += [1] * padding
|
||||
|
||||
if return_token_type_ids:
|
||||
encoding_dict["token_type_ids"][i] += [1] * padding
|
||||
|
||||
# Prepare inputs as tensors if asked
|
||||
if return_tensors == "tf" and is_tf_available():
|
||||
encoding_dict["input_ids"] = tf.constant([encoding_dict["input_ids"]])
|
||||
encoding_dict["input_ids"] = tf.constant(encoding_dict["input_ids"])
|
||||
if "token_type_ids" in encoding_dict:
|
||||
encoding_dict["token_type_ids"] = tf.constant([encoding_dict["token_type_ids"]])
|
||||
encoding_dict["token_type_ids"] = tf.constant(encoding_dict["token_type_ids"])
|
||||
|
||||
if "attention_mask" in encoding_dict:
|
||||
encoding_dict["attention_mask"] = tf.constant([encoding_dict["attention_mask"]])
|
||||
encoding_dict["attention_mask"] = tf.constant(encoding_dict["attention_mask"])
|
||||
|
||||
elif return_tensors == "pt" and is_torch_available():
|
||||
encoding_dict["input_ids"] = torch.tensor([encoding_dict["input_ids"]])
|
||||
encoding_dict["input_ids"] = torch.tensor(encoding_dict["input_ids"])
|
||||
if "token_type_ids" in encoding_dict:
|
||||
encoding_dict["token_type_ids"] = torch.tensor([encoding_dict["token_type_ids"]])
|
||||
encoding_dict["token_type_ids"] = torch.tensor(encoding_dict["token_type_ids"])
|
||||
|
||||
if "attention_mask" in encoding_dict:
|
||||
encoding_dict["attention_mask"] = torch.tensor([encoding_dict["attention_mask"]])
|
||||
encoding_dict["attention_mask"] = torch.tensor(encoding_dict["attention_mask"])
|
||||
elif return_tensors is not None:
|
||||
logger.warning(
|
||||
"Unable to convert output to tensors format {}, PyTorch or TensorFlow is not available.".format(
|
||||
@@ -1532,73 +1608,110 @@ class PreTrainedTokenizerFast(PreTrainedTokenizer):
|
||||
)
|
||||
)
|
||||
|
||||
return encoding_dict
|
||||
|
||||
def encode_plus(
|
||||
self,
|
||||
text,
|
||||
text_pair=None,
|
||||
return_tensors=None,
|
||||
return_token_type_ids=True,
|
||||
return_attention_mask=True,
|
||||
return_overflowing_tokens=False,
|
||||
return_special_tokens_mask=False,
|
||||
**kwargs
|
||||
):
|
||||
encoding = self.tokenizer.encode(text, text_pair)
|
||||
return self._convert_encoding(
|
||||
encoding,
|
||||
return_tensors=return_tensors,
|
||||
return_token_type_ids=return_token_type_ids,
|
||||
return_attention_mask=return_attention_mask,
|
||||
return_overflowing_tokens=return_overflowing_tokens,
|
||||
return_special_tokens_mask=return_special_tokens_mask,
|
||||
)
|
||||
|
||||
def tokenize(self, text):
|
||||
return self.tokenizer.encode(text).tokens
|
||||
return {k: v if len(v) > 1 else v[0] for k, v in encoding_dict.items()}
|
||||
|
||||
def _convert_token_to_id_with_added_voc(self, token):
|
||||
id = self.tokenizer.token_to_id(token)
|
||||
id = self._tokenizer.token_to_id(token)
|
||||
if id is None:
|
||||
return self.unk_token_id
|
||||
return id
|
||||
|
||||
def _convert_id_to_token(self, index):
|
||||
return self.tokenizer.id_to_token(int(index))
|
||||
return self._tokenizer.id_to_token(int(index))
|
||||
|
||||
def convert_tokens_to_string(self, tokens):
|
||||
return self.decoder.decode(tokens)
|
||||
return self._tokenizer.decode(tokens)
|
||||
|
||||
def add_tokens(self, new_tokens):
|
||||
self.tokenizer.add_tokens(new_tokens)
|
||||
self._tokenizer.add_tokens(new_tokens)
|
||||
|
||||
def add_special_tokens(self, special_tokens_dict):
|
||||
added = super().add_special_tokens(special_tokens_dict)
|
||||
self._update_special_tokens()
|
||||
return added
|
||||
|
||||
def encode_batch(
|
||||
def encode_plus(
|
||||
self,
|
||||
texts,
|
||||
text,
|
||||
text_pair=None,
|
||||
add_special_tokens=True,
|
||||
max_length=None,
|
||||
stride=0,
|
||||
truncation_strategy="longest_first",
|
||||
pad_to_max_length=False,
|
||||
return_tensors=None,
|
||||
return_token_type_ids=True,
|
||||
return_attention_mask=True,
|
||||
return_overflowing_tokens=False,
|
||||
return_special_tokens_mask=False,
|
||||
return_offsets_mapping=False,
|
||||
**kwargs
|
||||
):
|
||||
return [
|
||||
# Ensure we have text defined as [str]
|
||||
if text is not None and not isinstance(text, list):
|
||||
text = [text]
|
||||
|
||||
if text_pair is not None and not isinstance(text_pair, list):
|
||||
text_pair = [text_pair]
|
||||
|
||||
# Ensure we have all the pairs
|
||||
if len(text_pair) != len(text):
|
||||
raise ValueError(
|
||||
"Number of text_pair ({}) doesn't match number of text ({})".format(len(text_pair), len(text))
|
||||
)
|
||||
|
||||
# Set the truncation and padding strategy and restore the initial configuration
|
||||
with truncate_and_pad(
|
||||
self._tokenizer,
|
||||
max_length,
|
||||
stride,
|
||||
truncation_strategy,
|
||||
pad_to_max_length,
|
||||
self.padding_side,
|
||||
self.pad_token_id,
|
||||
self.pad_token_type_id,
|
||||
self._pad_token,
|
||||
):
|
||||
|
||||
if text_pair is None:
|
||||
tokens = self._tokenizer.encode_batch(text)
|
||||
else:
|
||||
tokens = self._tokenizer.encode_batch(list(zip(text, text_pair)))
|
||||
|
||||
# Convert encoding to dict
|
||||
max_length = max(map(lambda e: len(e.ids), tokens))
|
||||
tokens = [
|
||||
self._convert_encoding(
|
||||
encoding,
|
||||
return_tensors=return_tensors,
|
||||
return_token_type_ids=return_token_type_ids,
|
||||
return_attention_mask=return_attention_mask,
|
||||
return_overflowing_tokens=return_overflowing_tokens,
|
||||
return_special_tokens_mask=return_special_tokens_mask,
|
||||
return_tensors,
|
||||
return_token_type_ids,
|
||||
return_attention_mask,
|
||||
return_overflowing_tokens,
|
||||
return_special_tokens_mask,
|
||||
return_offsets_mapping,
|
||||
self.pad_token_id,
|
||||
max_length,
|
||||
)
|
||||
for encoding in self.tokenizer.encode_batch(texts)
|
||||
for encoding in tokens
|
||||
]
|
||||
|
||||
# Unwrap from the list if only on sample
|
||||
if len(tokens) == 1:
|
||||
return tokens[0]
|
||||
|
||||
# Sanitize the output to have dict[list] from list[dict]
|
||||
sanitized = {}
|
||||
for key in tokens[0].keys():
|
||||
stack = [item[key] for item in tokens]
|
||||
|
||||
if return_tensors == "tf":
|
||||
stack = tf.concat(stack, axis=0)
|
||||
elif return_tensors == "pt":
|
||||
stack = torch.cat(stack, dim=0)
|
||||
|
||||
sanitized[key] = stack
|
||||
return sanitized
|
||||
|
||||
def decode(self, token_ids, skip_special_tokens=False, clean_up_tokenization_spaces=True):
|
||||
text = self.tokenizer.decode(token_ids, skip_special_tokens)
|
||||
|
||||
@@ -1608,8 +1721,5 @@ class PreTrainedTokenizerFast(PreTrainedTokenizer):
|
||||
else:
|
||||
return text
|
||||
|
||||
def decode_batch(self, ids_batch, skip_special_tokens=False, clear_up_tokenization_spaces=True):
|
||||
return [
|
||||
self.clean_up_tokenization(text) if clear_up_tokenization_spaces else text
|
||||
for text in self.tokenizer.decode_batch(ids_batch, skip_special_tokens)
|
||||
]
|
||||
def save_vocabulary(self, save_directory):
|
||||
self._tokenizer.save(save_directory)
|
||||
|
||||
@@ -176,7 +176,10 @@ class XLMRobertaTokenizer(PreTrainedTokenizer):
|
||||
def create_token_type_ids_from_sequences(self, token_ids_0, token_ids_1=None):
|
||||
"""
|
||||
Creates a mask from the two sequences passed to be used in a sequence-pair classification task.
|
||||
RoBERTa does not make use of token type ids, therefore a list of zeros is returned.
|
||||
A RoBERTa sequence pair mask has the following format:
|
||||
0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1
|
||||
| first sequence | second sequence
|
||||
|
||||
if token_ids_1 is None, only returns the first portion of the mask (0's).
|
||||
"""
|
||||
sep = [self.sep_token_id]
|
||||
@@ -184,7 +187,7 @@ class XLMRobertaTokenizer(PreTrainedTokenizer):
|
||||
|
||||
if token_ids_1 is None:
|
||||
return len(cls + token_ids_0 + sep) * [0]
|
||||
return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]
|
||||
return len(cls + token_ids_0 + sep + sep) * [0] + len(token_ids_1 + sep) * [1]
|
||||
|
||||
@property
|
||||
def vocab_size(self):
|
||||
|
||||
@@ -333,8 +333,7 @@ def evaluate(args, model, tokenizer, prefix=""):
|
||||
|
||||
def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False):
|
||||
if args.local_rank not in [-1, 0] and not evaluate:
|
||||
torch.distributed.barrier() # Make sure only the first process in distributed training process the dataset,
|
||||
# and the others will use the cache
|
||||
torch.distributed.barrier() # Make sure only the first process in distributed training process the dataset, and the others will use the cache
|
||||
|
||||
# Load data features from cache or dataset file
|
||||
input_file = args.predict_file if evaluate else args.train_file
|
||||
@@ -367,8 +366,7 @@ def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=Fal
|
||||
torch.save(features, cached_features_file)
|
||||
|
||||
if args.local_rank == 0 and not evaluate:
|
||||
torch.distributed.barrier() # Make sure only the first process in distributed training process the dataset,
|
||||
# and the others will use the cache
|
||||
torch.distributed.barrier() # Make sure only the first process in distributed training process the dataset, and the others will use the cache
|
||||
|
||||
# Convert to Tensors and build dataset
|
||||
all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
|
||||
@@ -622,8 +620,7 @@ def main():
|
||||
|
||||
# Load pretrained model and tokenizer
|
||||
if args.local_rank not in [-1, 0]:
|
||||
torch.distributed.barrier() # Make sure only the first process in distributed training will
|
||||
# download model & vocab
|
||||
torch.distributed.barrier() # Make sure only the first process in distributed training will download model & vocab
|
||||
|
||||
args.model_type = args.model_type.lower()
|
||||
config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
|
||||
@@ -644,16 +641,15 @@ def main():
|
||||
)
|
||||
|
||||
if args.local_rank == 0:
|
||||
torch.distributed.barrier() # Make sure only the first process in distributed training will
|
||||
# download model & vocab
|
||||
torch.distributed.barrier() # Make sure only the first process in distributed training will download model & vocab
|
||||
|
||||
model.to(args.device)
|
||||
|
||||
logger.info("Training/evaluation parameters %s", args)
|
||||
|
||||
# Before we do anything with models, we want to ensure that we get fp16 execution of torch.einsum
|
||||
# if args.fp16 is set. Otherwise it'll default to "promote" mode, and we'll get fp32 operations.
|
||||
# Note that running `--fp16_opt_level="O2"` will remove the need for this code, but it is still valid.
|
||||
# Before we do anything with models, we want to ensure that we get fp16 execution of torch.einsum if args.fp16 is set.
|
||||
# Otherwise it'll default to "promote" mode, and we'll get fp32 operations. Note that running `--fp16_opt_level="O2"` will
|
||||
# remove the need for this code, but it is still valid.
|
||||
if args.fp16:
|
||||
try:
|
||||
import apex
|
||||
|
||||
@@ -28,8 +28,6 @@ if is_torch_available():
|
||||
BertConfig,
|
||||
AutoModel,
|
||||
BertModel,
|
||||
AutoModelForPreTraining,
|
||||
BertForPreTraining,
|
||||
AutoModelWithLMHead,
|
||||
BertForMaskedLM,
|
||||
RobertaForMaskedLM,
|
||||
@@ -39,14 +37,6 @@ if is_torch_available():
|
||||
BertForQuestionAnswering,
|
||||
)
|
||||
from transformers.modeling_bert import BERT_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
from transformers.modeling_auto import (
|
||||
MODEL_MAPPING,
|
||||
MODEL_FOR_PRETRAINING_MAPPING,
|
||||
MODEL_FOR_QUESTION_ANSWERING_MAPPING,
|
||||
MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING,
|
||||
MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING,
|
||||
MODEL_WITH_LM_HEAD_MAPPING,
|
||||
)
|
||||
|
||||
|
||||
@require_torch
|
||||
@@ -66,21 +56,6 @@ class AutoModelTest(unittest.TestCase):
|
||||
for value in loading_info.values():
|
||||
self.assertEqual(len(value), 0)
|
||||
|
||||
@slow
|
||||
def test_model_for_pretraining_from_pretrained(self):
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
for model_name in list(BERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
|
||||
config = AutoConfig.from_pretrained(model_name)
|
||||
self.assertIsNotNone(config)
|
||||
self.assertIsInstance(config, BertConfig)
|
||||
|
||||
model = AutoModelForPreTraining.from_pretrained(model_name)
|
||||
model, loading_info = AutoModelForPreTraining.from_pretrained(model_name, output_loading_info=True)
|
||||
self.assertIsNotNone(model)
|
||||
self.assertIsInstance(model, BertForPreTraining)
|
||||
for value in loading_info.values():
|
||||
self.assertEqual(len(value), 0)
|
||||
|
||||
@slow
|
||||
def test_lmhead_model_from_pretrained(self):
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
@@ -135,26 +110,3 @@ class AutoModelTest(unittest.TestCase):
|
||||
self.assertIsInstance(model, RobertaForMaskedLM)
|
||||
self.assertEqual(model.num_parameters(), 14830)
|
||||
self.assertEqual(model.num_parameters(only_trainable=True), 14830)
|
||||
|
||||
def test_parents_and_children_in_mappings(self):
|
||||
# Test that the children are placed before the parents in the mappings, as the `instanceof` will be triggered
|
||||
# by the parents and will return the wrong configuration type when using auto models
|
||||
|
||||
mappings = (
|
||||
MODEL_MAPPING,
|
||||
MODEL_FOR_PRETRAINING_MAPPING,
|
||||
MODEL_FOR_QUESTION_ANSWERING_MAPPING,
|
||||
MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING,
|
||||
MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING,
|
||||
MODEL_WITH_LM_HEAD_MAPPING,
|
||||
)
|
||||
|
||||
for mapping in mappings:
|
||||
mapping = tuple(mapping.items())
|
||||
for index, (child_config, child_model) in enumerate(mapping[1:]):
|
||||
for parent_config, parent_model in mapping[: index + 1]:
|
||||
with self.subTest(
|
||||
msg="Testing if {} is child of {}".format(child_config.__name__, parent_config.__name__)
|
||||
):
|
||||
self.assertFalse(issubclass(child_config, parent_config))
|
||||
self.assertFalse(issubclass(child_model, parent_model))
|
||||
|
||||
@@ -117,11 +117,23 @@ class ModelTesterMixin:
|
||||
|
||||
def test_attention_outputs(self):
|
||||
config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
|
||||
seq_len = getattr(self.model_tester, "seq_length", None)
|
||||
decoder_seq_length = getattr(self.model_tester, "decoder_seq_length", seq_len)
|
||||
encoder_seq_length = getattr(self.model_tester, "encoder_seq_length", seq_len)
|
||||
decoder_key_length = getattr(self.model_tester, "key_length", decoder_seq_length)
|
||||
encoder_key_length = getattr(self.model_tester, "key_length", encoder_seq_length)
|
||||
|
||||
decoder_seq_length = (
|
||||
self.model_tester.decoder_seq_length
|
||||
if hasattr(self.model_tester, "decoder_seq_length")
|
||||
else self.model_tester.seq_length
|
||||
)
|
||||
encoder_seq_length = (
|
||||
self.model_tester.encoder_seq_length
|
||||
if hasattr(self.model_tester, "encoder_seq_length")
|
||||
else self.model_tester.seq_length
|
||||
)
|
||||
decoder_key_length = (
|
||||
self.model_tester.key_length if hasattr(self.model_tester, "key_length") else decoder_seq_length
|
||||
)
|
||||
encoder_key_length = (
|
||||
self.model_tester.key_length if hasattr(self.model_tester, "key_length") else encoder_seq_length
|
||||
)
|
||||
|
||||
for model_class in self.all_model_classes:
|
||||
config.output_attentions = True
|
||||
|
||||
@@ -1,392 +0,0 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2018 The Google AI Language Team Authors.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
import unittest
|
||||
|
||||
from transformers import is_torch_available
|
||||
|
||||
from .test_configuration_common import ConfigTester
|
||||
from .test_modeling_common import ModelTesterMixin, ids_tensor
|
||||
from .utils import CACHE_DIR, require_torch, slow, torch_device
|
||||
|
||||
|
||||
if is_torch_available():
|
||||
from transformers import (
|
||||
FlaubertConfig,
|
||||
FlaubertModel,
|
||||
FlaubertWithLMHeadModel,
|
||||
FlaubertForQuestionAnswering,
|
||||
FlaubertForQuestionAnsweringSimple,
|
||||
FlaubertForSequenceClassification,
|
||||
)
|
||||
from transformers.modeling_flaubert import FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
|
||||
@require_torch
|
||||
class FlaubertModelTest(ModelTesterMixin, unittest.TestCase):
|
||||
|
||||
all_model_classes = (
|
||||
(
|
||||
FlaubertModel,
|
||||
FlaubertWithLMHeadModel,
|
||||
FlaubertForQuestionAnswering,
|
||||
FlaubertForQuestionAnsweringSimple,
|
||||
FlaubertForSequenceClassification,
|
||||
)
|
||||
if is_torch_available()
|
||||
else ()
|
||||
)
|
||||
|
||||
class FlaubertModelTester(object):
|
||||
def __init__(
|
||||
self,
|
||||
parent,
|
||||
batch_size=13,
|
||||
seq_length=7,
|
||||
is_training=True,
|
||||
use_input_lengths=True,
|
||||
use_token_type_ids=True,
|
||||
use_labels=True,
|
||||
gelu_activation=True,
|
||||
sinusoidal_embeddings=False,
|
||||
causal=False,
|
||||
asm=False,
|
||||
n_langs=2,
|
||||
vocab_size=99,
|
||||
n_special=0,
|
||||
hidden_size=32,
|
||||
num_hidden_layers=5,
|
||||
num_attention_heads=4,
|
||||
hidden_dropout_prob=0.1,
|
||||
attention_probs_dropout_prob=0.1,
|
||||
max_position_embeddings=512,
|
||||
type_vocab_size=16,
|
||||
type_sequence_label_size=2,
|
||||
initializer_range=0.02,
|
||||
num_labels=3,
|
||||
num_choices=4,
|
||||
summary_type="last",
|
||||
use_proj=True,
|
||||
scope=None,
|
||||
):
|
||||
self.parent = parent
|
||||
self.batch_size = batch_size
|
||||
self.seq_length = seq_length
|
||||
self.is_training = is_training
|
||||
self.use_input_lengths = use_input_lengths
|
||||
self.use_token_type_ids = use_token_type_ids
|
||||
self.use_labels = use_labels
|
||||
self.gelu_activation = gelu_activation
|
||||
self.sinusoidal_embeddings = sinusoidal_embeddings
|
||||
self.asm = asm
|
||||
self.n_langs = n_langs
|
||||
self.vocab_size = vocab_size
|
||||
self.n_special = n_special
|
||||
self.summary_type = summary_type
|
||||
self.causal = causal
|
||||
self.use_proj = use_proj
|
||||
self.hidden_size = hidden_size
|
||||
self.num_hidden_layers = num_hidden_layers
|
||||
self.num_attention_heads = num_attention_heads
|
||||
self.hidden_dropout_prob = hidden_dropout_prob
|
||||
self.attention_probs_dropout_prob = attention_probs_dropout_prob
|
||||
self.max_position_embeddings = max_position_embeddings
|
||||
self.n_langs = n_langs
|
||||
self.type_sequence_label_size = type_sequence_label_size
|
||||
self.initializer_range = initializer_range
|
||||
self.summary_type = summary_type
|
||||
self.num_labels = num_labels
|
||||
self.num_choices = num_choices
|
||||
self.scope = scope
|
||||
|
||||
def prepare_config_and_inputs(self):
|
||||
input_ids = ids_tensor([self.batch_size, self.seq_length], self.vocab_size)
|
||||
input_mask = ids_tensor([self.batch_size, self.seq_length], 2).float()
|
||||
|
||||
input_lengths = None
|
||||
if self.use_input_lengths:
|
||||
input_lengths = (
|
||||
ids_tensor([self.batch_size], vocab_size=2) + self.seq_length - 2
|
||||
) # small variation of seq_length
|
||||
|
||||
token_type_ids = None
|
||||
if self.use_token_type_ids:
|
||||
token_type_ids = ids_tensor([self.batch_size, self.seq_length], self.n_langs)
|
||||
|
||||
sequence_labels = None
|
||||
token_labels = None
|
||||
is_impossible_labels = None
|
||||
if self.use_labels:
|
||||
sequence_labels = ids_tensor([self.batch_size], self.type_sequence_label_size)
|
||||
token_labels = ids_tensor([self.batch_size, self.seq_length], self.num_labels)
|
||||
is_impossible_labels = ids_tensor([self.batch_size], 2).float()
|
||||
|
||||
config = FlaubertConfig(
|
||||
vocab_size=self.vocab_size,
|
||||
n_special=self.n_special,
|
||||
emb_dim=self.hidden_size,
|
||||
n_layers=self.num_hidden_layers,
|
||||
n_heads=self.num_attention_heads,
|
||||
dropout=self.hidden_dropout_prob,
|
||||
attention_dropout=self.attention_probs_dropout_prob,
|
||||
gelu_activation=self.gelu_activation,
|
||||
sinusoidal_embeddings=self.sinusoidal_embeddings,
|
||||
asm=self.asm,
|
||||
causal=self.causal,
|
||||
n_langs=self.n_langs,
|
||||
max_position_embeddings=self.max_position_embeddings,
|
||||
initializer_range=self.initializer_range,
|
||||
summary_type=self.summary_type,
|
||||
use_proj=self.use_proj,
|
||||
)
|
||||
|
||||
return (
|
||||
config,
|
||||
input_ids,
|
||||
token_type_ids,
|
||||
input_lengths,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
is_impossible_labels,
|
||||
input_mask,
|
||||
)
|
||||
|
||||
def check_loss_output(self, result):
|
||||
self.parent.assertListEqual(list(result["loss"].size()), [])
|
||||
|
||||
def create_and_check_flaubert_model(
|
||||
self,
|
||||
config,
|
||||
input_ids,
|
||||
token_type_ids,
|
||||
input_lengths,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
is_impossible_labels,
|
||||
input_mask,
|
||||
):
|
||||
model = FlaubertModel(config=config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
outputs = model(input_ids, lengths=input_lengths, langs=token_type_ids)
|
||||
outputs = model(input_ids, langs=token_type_ids)
|
||||
outputs = model(input_ids)
|
||||
sequence_output = outputs[0]
|
||||
result = {
|
||||
"sequence_output": sequence_output,
|
||||
}
|
||||
self.parent.assertListEqual(
|
||||
list(result["sequence_output"].size()), [self.batch_size, self.seq_length, self.hidden_size]
|
||||
)
|
||||
|
||||
def create_and_check_flaubert_lm_head(
|
||||
self,
|
||||
config,
|
||||
input_ids,
|
||||
token_type_ids,
|
||||
input_lengths,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
is_impossible_labels,
|
||||
input_mask,
|
||||
):
|
||||
model = FlaubertWithLMHeadModel(config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
|
||||
loss, logits = model(input_ids, token_type_ids=token_type_ids, labels=token_labels)
|
||||
|
||||
result = {
|
||||
"loss": loss,
|
||||
"logits": logits,
|
||||
}
|
||||
|
||||
self.parent.assertListEqual(list(result["loss"].size()), [])
|
||||
self.parent.assertListEqual(
|
||||
list(result["logits"].size()), [self.batch_size, self.seq_length, self.vocab_size]
|
||||
)
|
||||
|
||||
def create_and_check_flaubert_simple_qa(
|
||||
self,
|
||||
config,
|
||||
input_ids,
|
||||
token_type_ids,
|
||||
input_lengths,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
is_impossible_labels,
|
||||
input_mask,
|
||||
):
|
||||
model = FlaubertForQuestionAnsweringSimple(config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
|
||||
outputs = model(input_ids)
|
||||
|
||||
outputs = model(input_ids, start_positions=sequence_labels, end_positions=sequence_labels)
|
||||
loss, start_logits, end_logits = outputs
|
||||
|
||||
result = {
|
||||
"loss": loss,
|
||||
"start_logits": start_logits,
|
||||
"end_logits": end_logits,
|
||||
}
|
||||
self.parent.assertListEqual(list(result["start_logits"].size()), [self.batch_size, self.seq_length])
|
||||
self.parent.assertListEqual(list(result["end_logits"].size()), [self.batch_size, self.seq_length])
|
||||
self.check_loss_output(result)
|
||||
|
||||
def create_and_check_flaubert_qa(
|
||||
self,
|
||||
config,
|
||||
input_ids,
|
||||
token_type_ids,
|
||||
input_lengths,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
is_impossible_labels,
|
||||
input_mask,
|
||||
):
|
||||
model = FlaubertForQuestionAnswering(config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
|
||||
outputs = model(input_ids)
|
||||
start_top_log_probs, start_top_index, end_top_log_probs, end_top_index, cls_logits = outputs
|
||||
|
||||
outputs = model(
|
||||
input_ids,
|
||||
start_positions=sequence_labels,
|
||||
end_positions=sequence_labels,
|
||||
cls_index=sequence_labels,
|
||||
is_impossible=is_impossible_labels,
|
||||
p_mask=input_mask,
|
||||
)
|
||||
|
||||
outputs = model(
|
||||
input_ids,
|
||||
start_positions=sequence_labels,
|
||||
end_positions=sequence_labels,
|
||||
cls_index=sequence_labels,
|
||||
is_impossible=is_impossible_labels,
|
||||
)
|
||||
|
||||
(total_loss,) = outputs
|
||||
|
||||
outputs = model(input_ids, start_positions=sequence_labels, end_positions=sequence_labels)
|
||||
|
||||
(total_loss,) = outputs
|
||||
|
||||
result = {
|
||||
"loss": total_loss,
|
||||
"start_top_log_probs": start_top_log_probs,
|
||||
"start_top_index": start_top_index,
|
||||
"end_top_log_probs": end_top_log_probs,
|
||||
"end_top_index": end_top_index,
|
||||
"cls_logits": cls_logits,
|
||||
}
|
||||
|
||||
self.parent.assertListEqual(list(result["loss"].size()), [])
|
||||
self.parent.assertListEqual(
|
||||
list(result["start_top_log_probs"].size()), [self.batch_size, model.config.start_n_top]
|
||||
)
|
||||
self.parent.assertListEqual(
|
||||
list(result["start_top_index"].size()), [self.batch_size, model.config.start_n_top]
|
||||
)
|
||||
self.parent.assertListEqual(
|
||||
list(result["end_top_log_probs"].size()),
|
||||
[self.batch_size, model.config.start_n_top * model.config.end_n_top],
|
||||
)
|
||||
self.parent.assertListEqual(
|
||||
list(result["end_top_index"].size()),
|
||||
[self.batch_size, model.config.start_n_top * model.config.end_n_top],
|
||||
)
|
||||
self.parent.assertListEqual(list(result["cls_logits"].size()), [self.batch_size])
|
||||
|
||||
def create_and_check_flaubert_sequence_classif(
|
||||
self,
|
||||
config,
|
||||
input_ids,
|
||||
token_type_ids,
|
||||
input_lengths,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
is_impossible_labels,
|
||||
input_mask,
|
||||
):
|
||||
model = FlaubertForSequenceClassification(config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
|
||||
(logits,) = model(input_ids)
|
||||
loss, logits = model(input_ids, labels=sequence_labels)
|
||||
|
||||
result = {
|
||||
"loss": loss,
|
||||
"logits": logits,
|
||||
}
|
||||
|
||||
self.parent.assertListEqual(list(result["loss"].size()), [])
|
||||
self.parent.assertListEqual(
|
||||
list(result["logits"].size()), [self.batch_size, self.type_sequence_label_size]
|
||||
)
|
||||
|
||||
def prepare_config_and_inputs_for_common(self):
|
||||
config_and_inputs = self.prepare_config_and_inputs()
|
||||
(
|
||||
config,
|
||||
input_ids,
|
||||
token_type_ids,
|
||||
input_lengths,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
is_impossible_labels,
|
||||
input_mask,
|
||||
) = config_and_inputs
|
||||
inputs_dict = {"input_ids": input_ids, "token_type_ids": token_type_ids, "lengths": input_lengths}
|
||||
return config, inputs_dict
|
||||
|
||||
def setUp(self):
|
||||
self.model_tester = FlaubertModelTest.FlaubertModelTester(self)
|
||||
self.config_tester = ConfigTester(self, config_class=FlaubertConfig, emb_dim=37)
|
||||
|
||||
def test_config(self):
|
||||
self.config_tester.run_common_tests()
|
||||
|
||||
def test_flaubert_model(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_flaubert_model(*config_and_inputs)
|
||||
|
||||
def test_flaubert_lm_head(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_flaubert_lm_head(*config_and_inputs)
|
||||
|
||||
def test_flaubert_simple_qa(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_flaubert_simple_qa(*config_and_inputs)
|
||||
|
||||
def test_flaubert_qa(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_flaubert_qa(*config_and_inputs)
|
||||
|
||||
def test_flaubert_sequence_classif(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_flaubert_sequence_classif(*config_and_inputs)
|
||||
|
||||
@slow
|
||||
def test_model_from_pretrained(self):
|
||||
for model_name in list(FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
|
||||
model = FlaubertModel.from_pretrained(model_name, cache_dir=CACHE_DIR)
|
||||
self.assertIsNotNone(model)
|
||||
@@ -32,7 +32,7 @@ if is_torch_available():
|
||||
RobertaForSequenceClassification,
|
||||
RobertaForTokenClassification,
|
||||
)
|
||||
from transformers.modeling_roberta import RobertaEmbeddings, RobertaForMultipleChoice, RobertaForQuestionAnswering
|
||||
from transformers.modeling_roberta import RobertaEmbeddings
|
||||
from transformers.modeling_roberta import ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
|
||||
|
||||
|
||||
@@ -184,51 +184,6 @@ class RobertaModelTest(ModelTesterMixin, unittest.TestCase):
|
||||
)
|
||||
self.check_loss_output(result)
|
||||
|
||||
def create_and_check_roberta_for_multiple_choice(
|
||||
self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels, choice_labels
|
||||
):
|
||||
config.num_choices = self.num_choices
|
||||
model = RobertaForMultipleChoice(config=config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
multiple_choice_inputs_ids = input_ids.unsqueeze(1).expand(-1, self.num_choices, -1).contiguous()
|
||||
multiple_choice_token_type_ids = token_type_ids.unsqueeze(1).expand(-1, self.num_choices, -1).contiguous()
|
||||
multiple_choice_input_mask = input_mask.unsqueeze(1).expand(-1, self.num_choices, -1).contiguous()
|
||||
loss, logits = model(
|
||||
multiple_choice_inputs_ids,
|
||||
attention_mask=multiple_choice_input_mask,
|
||||
token_type_ids=multiple_choice_token_type_ids,
|
||||
labels=choice_labels,
|
||||
)
|
||||
result = {
|
||||
"loss": loss,
|
||||
"logits": logits,
|
||||
}
|
||||
self.parent.assertListEqual(list(result["logits"].size()), [self.batch_size, self.num_choices])
|
||||
self.check_loss_output(result)
|
||||
|
||||
def create_and_check_roberta_for_question_answering(
|
||||
self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels, choice_labels
|
||||
):
|
||||
model = RobertaForQuestionAnswering(config=config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
loss, start_logits, end_logits = model(
|
||||
input_ids,
|
||||
attention_mask=input_mask,
|
||||
token_type_ids=token_type_ids,
|
||||
start_positions=sequence_labels,
|
||||
end_positions=sequence_labels,
|
||||
)
|
||||
result = {
|
||||
"loss": loss,
|
||||
"start_logits": start_logits,
|
||||
"end_logits": end_logits,
|
||||
}
|
||||
self.parent.assertListEqual(list(result["start_logits"].size()), [self.batch_size, self.seq_length])
|
||||
self.parent.assertListEqual(list(result["end_logits"].size()), [self.batch_size, self.seq_length])
|
||||
self.check_loss_output(result)
|
||||
|
||||
def prepare_config_and_inputs_for_common(self):
|
||||
config_and_inputs = self.prepare_config_and_inputs()
|
||||
(
|
||||
@@ -258,18 +213,6 @@ class RobertaModelTest(ModelTesterMixin, unittest.TestCase):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_roberta_for_masked_lm(*config_and_inputs)
|
||||
|
||||
def test_for_token_classification(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_roberta_for_token_classification(*config_and_inputs)
|
||||
|
||||
def test_for_multiple_choice(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_roberta_for_multiple_choice(*config_and_inputs)
|
||||
|
||||
def test_for_question_answering(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_roberta_for_question_answering(*config_and_inputs)
|
||||
|
||||
@slow
|
||||
def test_model_from_pretrained(self):
|
||||
for model_name in list(ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
|
||||
|
||||
@@ -28,8 +28,6 @@ if is_tf_available():
|
||||
BertConfig,
|
||||
TFAutoModel,
|
||||
TFBertModel,
|
||||
TFAutoModelForPreTraining,
|
||||
TFBertForPreTraining,
|
||||
TFAutoModelWithLMHead,
|
||||
TFBertForMaskedLM,
|
||||
TFRobertaForMaskedLM,
|
||||
@@ -59,23 +57,6 @@ class TFAutoModelTest(unittest.TestCase):
|
||||
self.assertIsNotNone(model)
|
||||
self.assertIsInstance(model, TFBertModel)
|
||||
|
||||
@slow
|
||||
def test_model_for_pretraining_from_pretrained(self):
|
||||
import h5py
|
||||
|
||||
self.assertTrue(h5py.version.hdf5_version.startswith("1.10"))
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
# for model_name in list(TF_BERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
|
||||
for model_name in ["bert-base-uncased"]:
|
||||
config = AutoConfig.from_pretrained(model_name)
|
||||
self.assertIsNotNone(config)
|
||||
self.assertIsInstance(config, BertConfig)
|
||||
|
||||
model = TFAutoModelForPreTraining.from_pretrained(model_name)
|
||||
self.assertIsNotNone(model)
|
||||
self.assertIsInstance(model, TFBertForPreTraining)
|
||||
|
||||
@slow
|
||||
def test_lmhead_model_from_pretrained(self):
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
@@ -112,15 +112,8 @@ class TFModelTesterMixin:
|
||||
tfo = tf_model(inputs_dict, training=False)
|
||||
tf_hidden_states = tfo[0].numpy()
|
||||
pt_hidden_states = pto[0].numpy()
|
||||
|
||||
tf_nans = np.copy(np.isnan(tf_hidden_states))
|
||||
pt_nans = np.copy(np.isnan(pt_hidden_states))
|
||||
|
||||
pt_hidden_states[tf_nans] = 0
|
||||
tf_hidden_states[tf_nans] = 0
|
||||
pt_hidden_states[pt_nans] = 0
|
||||
tf_hidden_states[pt_nans] = 0
|
||||
|
||||
tf_hidden_states[np.isnan(tf_hidden_states)] = 0
|
||||
pt_hidden_states[np.isnan(pt_hidden_states)] = 0
|
||||
max_diff = np.amax(np.abs(tf_hidden_states - pt_hidden_states))
|
||||
# Debug info (remove when fixed)
|
||||
if max_diff >= 2e-2:
|
||||
@@ -151,14 +144,8 @@ class TFModelTesterMixin:
|
||||
tfo = tf_model(inputs_dict)
|
||||
tfo = tfo[0].numpy()
|
||||
pto = pto[0].numpy()
|
||||
tf_nans = np.copy(np.isnan(tfo))
|
||||
pt_nans = np.copy(np.isnan(pto))
|
||||
|
||||
pto[tf_nans] = 0
|
||||
tfo[tf_nans] = 0
|
||||
pto[pt_nans] = 0
|
||||
tfo[pt_nans] = 0
|
||||
|
||||
tfo[np.isnan(tfo)] = 0
|
||||
pto[np.isnan(pto)] = 0
|
||||
max_diff = np.amax(np.abs(tfo - pto))
|
||||
self.assertLessEqual(max_diff, 2e-2)
|
||||
|
||||
|
||||
@@ -219,5 +219,5 @@ class TFDistilBertModelTest(TFModelTesterMixin, unittest.TestCase):
|
||||
# @slow
|
||||
# def test_model_from_pretrained(self):
|
||||
# for model_name in list(DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
|
||||
# model = DistilBertModesss.from_pretrained(model_name, cache_dir=CACHE_DIR)
|
||||
# model = DistilBertModel.from_pretrained(model_name, cache_dir=CACHE_DIR)
|
||||
# self.assertIsNotNone(model)
|
||||
|
||||
@@ -200,10 +200,6 @@ class TFRobertaModelTest(TFModelTesterMixin, unittest.TestCase):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_roberta_for_masked_lm(*config_and_inputs)
|
||||
|
||||
def test_for_token_classification(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_roberta_for_token_classification(*config_and_inputs)
|
||||
|
||||
@slow
|
||||
def test_model_from_pretrained(self):
|
||||
for model_name in list(TF_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
|
||||
|
||||
+8
-98
@@ -1,8 +1,7 @@
|
||||
import unittest
|
||||
from typing import Iterable, List, Optional
|
||||
from typing import Iterable
|
||||
|
||||
from transformers import pipeline
|
||||
from transformers.pipelines import Pipeline
|
||||
|
||||
from .utils import require_tf, require_torch
|
||||
|
||||
@@ -63,25 +62,9 @@ TEXT_CLASSIF_FINETUNED_MODELS = {
|
||||
)
|
||||
}
|
||||
|
||||
FILL_MASK_FINETUNED_MODELS = {
|
||||
("distilroberta-base", "distilroberta-base", None),
|
||||
}
|
||||
|
||||
TF_FILL_MASK_FINETUNED_MODELS = {
|
||||
("distilroberta-base", "distilroberta-base", None),
|
||||
}
|
||||
|
||||
|
||||
class MonoColumnInputTestCase(unittest.TestCase):
|
||||
def _test_mono_column_pipeline(
|
||||
self,
|
||||
nlp: Pipeline,
|
||||
valid_inputs: List,
|
||||
invalid_inputs: List,
|
||||
output_keys: Iterable[str],
|
||||
expected_multi_result: Optional[List] = None,
|
||||
expected_check_keys: Optional[List[str]] = None,
|
||||
):
|
||||
def _test_mono_column_pipeline(self, nlp, valid_inputs: list, invalid_inputs: list, output_keys: Iterable[str]):
|
||||
self.assertIsNotNone(nlp)
|
||||
|
||||
mono_result = nlp(valid_inputs[0])
|
||||
@@ -98,13 +81,6 @@ class MonoColumnInputTestCase(unittest.TestCase):
|
||||
self.assertIsInstance(multi_result, list)
|
||||
self.assertIsInstance(multi_result[0], (dict, list))
|
||||
|
||||
if expected_multi_result is not None:
|
||||
for result, expect in zip(multi_result, expected_multi_result):
|
||||
for key in expected_check_keys or []:
|
||||
self.assertEqual(
|
||||
set([o[key] for o in result]), set([o[key] for o in expect]),
|
||||
)
|
||||
|
||||
if isinstance(multi_result[0], list):
|
||||
multi_result = multi_result[0]
|
||||
|
||||
@@ -134,7 +110,7 @@ class MonoColumnInputTestCase(unittest.TestCase):
|
||||
|
||||
@require_torch
|
||||
def test_sentiment_analysis(self):
|
||||
mandatory_keys = {"label", "score"}
|
||||
mandatory_keys = {"label"}
|
||||
valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
|
||||
invalid_inputs = [None]
|
||||
for tokenizer, model, config in TEXT_CLASSIF_FINETUNED_MODELS:
|
||||
@@ -143,7 +119,7 @@ class MonoColumnInputTestCase(unittest.TestCase):
|
||||
|
||||
@require_tf
|
||||
def test_tf_sentiment_analysis(self):
|
||||
mandatory_keys = {"label", "score"}
|
||||
mandatory_keys = {"label"}
|
||||
valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
|
||||
invalid_inputs = [None]
|
||||
for tokenizer, model, config in TF_TEXT_CLASSIF_FINETUNED_MODELS:
|
||||
@@ -151,87 +127,21 @@ class MonoColumnInputTestCase(unittest.TestCase):
|
||||
self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)
|
||||
|
||||
@require_torch
|
||||
def test_feature_extraction(self):
|
||||
def test_features_extraction(self):
|
||||
valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
|
||||
invalid_inputs = [None]
|
||||
for tokenizer, model, config in FEATURE_EXTRACT_FINETUNED_MODELS:
|
||||
nlp = pipeline(task="feature-extraction", model=model, config=config, tokenizer=tokenizer)
|
||||
nlp = pipeline(task="sentiment-analysis", model=model, config=config, tokenizer=tokenizer)
|
||||
self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, {})
|
||||
|
||||
@require_tf
|
||||
def test_tf_feature_extraction(self):
|
||||
def test_tf_features_extraction(self):
|
||||
valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
|
||||
invalid_inputs = [None]
|
||||
for tokenizer, model, config in TF_FEATURE_EXTRACT_FINETUNED_MODELS:
|
||||
nlp = pipeline(task="feature-extraction", model=model, config=config, tokenizer=tokenizer)
|
||||
nlp = pipeline(task="sentiment-analysis", model=model, config=config, tokenizer=tokenizer)
|
||||
self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, {})
|
||||
|
||||
@require_torch
|
||||
def test_fill_mask(self):
|
||||
mandatory_keys = {"sequence", "score", "token"}
|
||||
valid_inputs = [
|
||||
"My name is <mask>",
|
||||
"The largest city in France is <mask>",
|
||||
]
|
||||
invalid_inputs = [None]
|
||||
expected_multi_result = [
|
||||
[
|
||||
{"score": 0.008698059245944023, "sequence": "<s>My name is John</s>", "token": 610},
|
||||
{"score": 0.007750614080578089, "sequence": "<s>My name is Chris</s>", "token": 1573},
|
||||
],
|
||||
[
|
||||
{"score": 0.2721288502216339, "sequence": "<s>The largest city in France is Paris</s>", "token": 2201},
|
||||
{
|
||||
"score": 0.19764970242977142,
|
||||
"sequence": "<s>The largest city in France is Lyon</s>",
|
||||
"token": 12790,
|
||||
},
|
||||
],
|
||||
]
|
||||
for tokenizer, model, config in FILL_MASK_FINETUNED_MODELS:
|
||||
nlp = pipeline(task="fill-mask", model=model, config=config, tokenizer=tokenizer, topk=2)
|
||||
self._test_mono_column_pipeline(
|
||||
nlp,
|
||||
valid_inputs,
|
||||
invalid_inputs,
|
||||
mandatory_keys,
|
||||
expected_multi_result=expected_multi_result,
|
||||
expected_check_keys=["sequence"],
|
||||
)
|
||||
|
||||
@require_tf
|
||||
def test_tf_fill_mask(self):
|
||||
mandatory_keys = {"sequence", "score", "token"}
|
||||
valid_inputs = [
|
||||
"My name is <mask>",
|
||||
"The largest city in France is <mask>",
|
||||
]
|
||||
invalid_inputs = [None]
|
||||
expected_multi_result = [
|
||||
[
|
||||
{"score": 0.008698059245944023, "sequence": "<s>My name is John</s>", "token": 610},
|
||||
{"score": 0.007750614080578089, "sequence": "<s>My name is Chris</s>", "token": 1573},
|
||||
],
|
||||
[
|
||||
{"score": 0.2721288502216339, "sequence": "<s>The largest city in France is Paris</s>", "token": 2201},
|
||||
{
|
||||
"score": 0.19764970242977142,
|
||||
"sequence": "<s>The largest city in France is Lyon</s>",
|
||||
"token": 12790,
|
||||
},
|
||||
],
|
||||
]
|
||||
for tokenizer, model, config in TF_FILL_MASK_FINETUNED_MODELS:
|
||||
nlp = pipeline(task="fill-mask", model=model, config=config, tokenizer=tokenizer, topk=2)
|
||||
self._test_mono_column_pipeline(
|
||||
nlp,
|
||||
valid_inputs,
|
||||
invalid_inputs,
|
||||
mandatory_keys,
|
||||
expected_multi_result=expected_multi_result,
|
||||
expected_check_keys=["sequence"],
|
||||
)
|
||||
|
||||
|
||||
class MultiColumnInputTestCase(unittest.TestCase):
|
||||
def _test_multicolumn_pipeline(self, nlp, valid_inputs: list, invalid_inputs: list, output_keys: Iterable[str]):
|
||||
|
||||
@@ -25,7 +25,6 @@ from transformers import (
|
||||
GPT2Tokenizer,
|
||||
RobertaTokenizer,
|
||||
)
|
||||
from transformers.tokenization_auto import TOKENIZER_MAPPING
|
||||
|
||||
from .utils import DUMMY_UNKWOWN_IDENTIFIER, SMALL_MODEL_IDENTIFIER, slow # noqa: F401
|
||||
|
||||
@@ -71,19 +70,3 @@ class AutoTokenizerTest(unittest.TestCase):
|
||||
for tokenizer_class in [BertTokenizer, AutoTokenizer]:
|
||||
with self.assertRaises(EnvironmentError):
|
||||
_ = tokenizer_class.from_pretrained("julien-c/herlolip-not-exists")
|
||||
|
||||
def test_parents_and_children_in_mappings(self):
|
||||
# Test that the children are placed before the parents in the mappings, as the `instanceof` will be triggered
|
||||
# by the parents and will return the wrong configuration type when using auto models
|
||||
|
||||
mappings = (TOKENIZER_MAPPING,)
|
||||
|
||||
for mapping in mappings:
|
||||
mapping = tuple(mapping.items())
|
||||
for index, (child_config, child_model) in enumerate(mapping[1:]):
|
||||
for parent_config, parent_model in mapping[: index + 1]:
|
||||
with self.subTest(
|
||||
msg="Testing if {} is child of {}".format(child_config.__name__, parent_config.__name__)
|
||||
):
|
||||
self.assertFalse(issubclass(child_config, parent_config))
|
||||
self.assertFalse(issubclass(child_model, parent_model))
|
||||
|
||||
@@ -495,16 +495,3 @@ class TokenizerTesterMixin:
|
||||
assert [token_type_padding_idx] * padding_size + token_type_ids == padded_token_type_ids
|
||||
assert [0] * padding_size + attention_mask == padded_attention_mask
|
||||
assert [1] * padding_size + special_tokens_mask == padded_special_tokens_mask
|
||||
|
||||
def test_separate_tokenizers(self):
|
||||
# This tests that tokenizers don't impact others. Unfortunately the case where it fails is when
|
||||
# we're loading an S3 configuration from a pre-trained identifier, and we have no way of testing those today.
|
||||
|
||||
tokenizer = self.get_tokenizer(random_argument=True)
|
||||
print(tokenizer.init_kwargs)
|
||||
assert tokenizer.init_kwargs["random_argument"] is True
|
||||
new_tokenizer = self.get_tokenizer(random_argument=False)
|
||||
print(tokenizer.init_kwargs)
|
||||
print(new_tokenizer.init_kwargs)
|
||||
assert tokenizer.init_kwargs["random_argument"] is True
|
||||
assert new_tokenizer.init_kwargs["random_argument"] is False
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
import unittest
|
||||
|
||||
from transformers import (
|
||||
BertTokenizer,
|
||||
BertTokenizerFast,
|
||||
CTRLTokenizer,
|
||||
DistilBertTokenizer,
|
||||
GPT2Tokenizer,
|
||||
GPT2TokenizerFast,
|
||||
OpenAIGPTTokenizer,
|
||||
RobertaTokenizer,
|
||||
)
|
||||
from transformers.tokenization_ctrl import CTRLTokenizerFast
|
||||
from transformers.tokenization_distilbert import DistilBertTokenizerFast
|
||||
from transformers.tokenization_openai import OpenAIGPTTokenizerFast
|
||||
from transformers.tokenization_roberta import RobertaTokenizerFast
|
||||
|
||||
|
||||
class FastTokenizerMatchingTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
with open("fixtures/sample_text.txt") as f_data:
|
||||
self._data = f_data.read()
|
||||
|
||||
def _tokenize_inputs_and_check_matching(self, tokenizer_p, tokenizer_r):
|
||||
# Ensure basic input match
|
||||
input_p = tokenizer_p.encode_plus(self._data)
|
||||
input_r = tokenizer_r.encode_plus(self._data)
|
||||
|
||||
self.assertSequenceEqual(input_p["input_ids"], input_r["input_ids"])
|
||||
self.assertSequenceEqual(input_p["token_type_ids"], input_r["token_type_ids"])
|
||||
self.assertSequenceEqual(input_p["attention_mask"], input_r["attention_mask"])
|
||||
|
||||
input_pairs_p = tokenizer_p.encode_plus(self._data, self._data)
|
||||
input_pairs_r = tokenizer_r.encode_plus(self._data, self._data)
|
||||
|
||||
self.assertSequenceEqual(input_pairs_p["input_ids"], input_pairs_r["input_ids"])
|
||||
self.assertSequenceEqual(input_pairs_p["token_type_ids"], input_pairs_r["token_type_ids"])
|
||||
self.assertSequenceEqual(input_pairs_p["attention_mask"], input_pairs_r["attention_mask"])
|
||||
|
||||
# Ensure truncation match
|
||||
input_p = tokenizer_p.encode_plus(self._data, max_length=512, pad_to_max_length=True)
|
||||
input_r = tokenizer_r.encode_plus(self._data, max_length=512, pad_to_max_length=True)
|
||||
|
||||
self.assertSequenceEqual(input_p["input_ids"], input_r["input_ids"])
|
||||
self.assertSequenceEqual(input_p["token_type_ids"], input_r["token_type_ids"])
|
||||
self.assertSequenceEqual(input_p["attention_mask"], input_r["attention_mask"])
|
||||
|
||||
# Ensure truncation with stride match
|
||||
# input_p = tokenizer_p.encode_plus(self._data, max_length=512, stride=3, return_overflowing_tokens=True)
|
||||
# input_r = tokenizer_r.encode_plus(self._data, max_length=512, stride=3, return_overflowing_tokens=True)
|
||||
#
|
||||
# self.assertSequenceEqual(input_p['input_ids'], input_r['input_ids'])
|
||||
# self.assertSequenceEqual(input_p['token_type_ids'], input_r['token_type_ids'])
|
||||
# self.assertSequenceEqual(input_p['attention_mask'], input_r['attention_mask'])
|
||||
|
||||
def test_bert(self):
|
||||
for tokenizer_name in BertTokenizer.pretrained_vocab_files_map["vocab_file"].keys():
|
||||
tokenizer_p = BertTokenizer.from_pretrained(tokenizer_name)
|
||||
tokenizer_r = BertTokenizerFast.from_pretrained(tokenizer_name)
|
||||
|
||||
self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
|
||||
|
||||
def test_ctrl(self):
|
||||
for tokenizer_name in CTRLTokenizer.pretrained_vocab_files_map["vocab_file"].keys():
|
||||
tokenizer_p = CTRLTokenizer.from_pretrained(tokenizer_name)
|
||||
tokenizer_r = CTRLTokenizerFast.from_pretrained(tokenizer_name)
|
||||
|
||||
self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
|
||||
|
||||
def test_distilbert(self):
|
||||
for tokenizer_name in DistilBertTokenizer.pretrained_vocab_files_map["vocab_file"].keys():
|
||||
tokenizer_p = DistilBertTokenizer.from_pretrained(tokenizer_name)
|
||||
tokenizer_r = DistilBertTokenizerFast.from_pretrained(tokenizer_name)
|
||||
|
||||
self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
|
||||
|
||||
def test_gpt2(self):
|
||||
for tokenizer_name in GPT2Tokenizer.pretrained_vocab_files_map["vocab_file"].keys():
|
||||
tokenizer_p = GPT2Tokenizer.from_pretrained(tokenizer_name)
|
||||
tokenizer_r = GPT2TokenizerFast.from_pretrained(tokenizer_name)
|
||||
|
||||
self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
|
||||
|
||||
def test_roberta(self):
|
||||
for tokenizer_name in RobertaTokenizer.pretrained_vocab_files_map["vocab_file"].keys():
|
||||
tokenizer_p = RobertaTokenizer.from_pretrained(tokenizer_name)
|
||||
tokenizer_r = RobertaTokenizerFast.from_pretrained(tokenizer_name)
|
||||
|
||||
self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
|
||||
|
||||
def test_openai(self):
|
||||
for tokenizer_name in OpenAIGPTTokenizer.pretrained_vocab_files_map["vocab_file"].keys():
|
||||
tokenizer_p = OpenAIGPTTokenizer.from_pretrained(tokenizer_name)
|
||||
tokenizer_r = OpenAIGPTTokenizerFast.from_pretrained(tokenizer_name)
|
||||
|
||||
self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+2
-4
@@ -1,12 +1,11 @@
|
||||
#!/usr/bin/env python
|
||||
from argparse import ArgumentParser
|
||||
|
||||
from transformers.commands.convert import ConvertCommand
|
||||
from transformers.commands.download import DownloadCommand
|
||||
from transformers.commands.env import EnvironmentCommand
|
||||
from transformers.commands.run import RunCommand
|
||||
from transformers.commands.serving import ServeCommand
|
||||
from transformers.commands.user import UserCommands
|
||||
from transformers.commands.convert import ConvertCommand
|
||||
from transformers.commands.serving import ServeCommand
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = ArgumentParser('Transformers CLI tool', usage='transformers-cli <command> [<args>]')
|
||||
@@ -15,7 +14,6 @@ if __name__ == '__main__':
|
||||
# Register commands
|
||||
ConvertCommand.register_subcommand(commands_parser)
|
||||
DownloadCommand.register_subcommand(commands_parser)
|
||||
EnvironmentCommand.register_subcommand(commands_parser)
|
||||
RunCommand.register_subcommand(commands_parser)
|
||||
ServeCommand.register_subcommand(commands_parser)
|
||||
UserCommands.register_subcommand(commands_parser)
|
||||
|
||||
Reference in New Issue
Block a user