updating docs - adding few tests to tokenizers
This commit is contained in:
1 parent
009273dbdd
commit
00132b7a7a
10 files changed
+390
-521
No files matched your search
@@ -1,6 +1,13 @@
|
||||
Models
|
||||
----------------------------------------------------
|
||||
|
||||
The base class ``PreTrainedModel`` implements the common methods for loading/saving a model either from a local file or directory, or from a pretrained model configuration provided by the library (downloaded from HuggingFace's AWS S3 repository).
|
||||
|
||||
``PreTrainedModel`` also implements a few methods which are common among all the models to:
|
||||
|
||||
- resize the input token embeddings when new tokens are added to the vocabulary
|
||||
- prune the attention heads of the model.
|
||||
|
||||
``PreTrainedModel``
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
|
||||
Reference in new issue
Block a user