big doc update [WIP]
This commit is contained in:
1 parent
bfbe52ec39
commit
009273dbdd
19 files changed
+189
-60
No files matched your search
@@ -1,17 +1,58 @@
|
||||
# Quickstart
|
||||
|
||||
## Philosophy
|
||||
|
||||
PyTorch-Transformers is an opinionated library built for NLP researchers seeking to use/study/extend large-scale transformers models.
|
||||
|
||||
The library was designed with two strong goals in mind:
|
||||
|
||||
- be as easy and fast to use as possible:
|
||||
|
||||
- we strongly limited the number of abstractions to learn, in fact there are almost no abstractions, just three standard classes for each model: configuration, models and tokenizer,
|
||||
- each pretrained model configuration, weights and vocabulary can be downloaded, cached and loaded in the related class in a simple way by using a common `from_pretrained()` instantiation method.
|
||||
- this library is NOT a modular toolbox of building blocks for neural nets, to extend/build-upon the library, just use your regular Python/PyTorch modules and inherit from the base classes of the library to reuse functionalities like model loading/saving.
|
||||
|
||||
- provide state-of-the-art models with performances as close as possible to the original models:
|
||||
|
||||
- we provide at least one example for each model which reproduces a result provided by the official authors of said model,
|
||||
- the code is usually as close to the original code base as possible which means some PyTorch code may be not as *pytorchic* as it could be as a result of being converted TensorFlow code.
|
||||
|
||||
A few other goals:
|
||||
|
||||
- expose the models internals as consistently as possible:
|
||||
|
||||
- we give access, using a single API to the full hidden-states and attention weights,
|
||||
- tokenizer and base model's API are standardized to easily switch between models.
|
||||
|
||||
- incorporate a subjective selection of promising tools for fine-tuning/investiguating these models:
|
||||
|
||||
- a simple/consistent way to add new tokens to the vocabulary and embeddings for fine-tuning,
|
||||
- simple ways to mask and prune transformer heads.
|
||||
|
||||
## Main concepts
|
||||
|
||||
The library is build around three type of classes for each models:
|
||||
|
||||
- **model classes** which are PyTorch models (`torch.nn.Modules`) of the 6 models architectures currently provided in the library, e.g. `BertModel`
|
||||
- **configuration classes** which store all the parameters required to build a model, e.g. `BertConfig`
|
||||
- **tokenizer classes** which store the vocabulary for each model and provide methods for encoding strings in list of token embeddings indices to be fed to a model, e.g. `BertTokenizer`
|
||||
|
||||
All these classes can be instantiated from pretrained instances and saved locally using two methods:
|
||||
|
||||
- `from_pretrained()` let you instantiate a model/configuration/tokenizer from a pretrained version either provided by the library itself (currently 27 models are provided as listed [here](https://huggingface.co/pytorch-transformers/pretrained_models.html)) or stored locally (or on a server) by the user,
|
||||
- `save_pretrained()` let you save a model/configuration/tokenizer locally so that it can be reloaded using `from_pretrained()`.
|
||||
|
||||
Let's go through a few simple quick-start examples to see how we can instantiate and use these classes.
|
||||
|
||||
## Quick tour: Usage
|
||||
|
||||
Here are two quick-start examples showcasing a few `Bert` and `GPT2` classes and pre-trained models.
|
||||
Here are two examples showcasing a few `Bert` and `GPT2` classes and pre-trained models.
|
||||
|
||||
See package reference for examples for each model classe.
|
||||
See full API reference for examples for each model classe.
|
||||
|
||||
### BERT example
|
||||
|
||||
First let's prepare a tokenized input from a text string using `BertTokenizer`
|
||||
Let's start by preparing a tokenized input (a list of token embeddings indices to be fed to Bert) from a text string using `BertTokenizer`
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
Reference in new issue
Block a user