Julien Chaumond
1748fdf657
[doc] Fix rst table
2019-12-11 18:32:27 -05:00
Julien Chaumond
36fc52a3b4
Update links to weights
2019-12-11 18:32:27 -05:00
Julien Chaumond
371c5ddfad
Py2 tests for Lysandre
2019-12-11 18:32:27 -05:00
Julien Chaumond
5505cf7014
Run tests on Py2 too, for Lysandre
2019-12-11 18:32:27 -05:00
Julien Chaumond
9cb97c0c0f
Actually run the tests
2019-12-11 18:32:27 -05:00
Julien Chaumond
95854c4a2f
Actually run the tests
2019-12-11 18:32:27 -05:00
Julien Chaumond
d2100428d3
Update to new test infra and only run conditionally
2019-12-11 18:32:27 -05:00
Masatoshi Suzuki
597ba7feb3
Support testing Japanese BERT tokenizers
2019-12-11 18:32:27 -05:00
Masatoshi Suzuki
6a43dc9d7d
Support Python 2
2019-12-11 18:32:27 -05:00
Masatoshi Suzuki
a09da4eeb0
Add a test for Japanese BERT tokenizers
2019-12-11 18:32:27 -05:00
Masatoshi Suzuki
57b5cb3eaa
Fix loading BertJapaneseTokenizer
2019-12-11 18:32:27 -05:00
Masatoshi Suzuki
c03c0dfd23
Add support for Japanese BERT models by cl-tohoku
2019-12-11 18:32:27 -05:00
Stefan Schweter
030faccb8d
doc: fix pretrained models table
2019-12-11 12:19:21 -05:00
Julien Chaumond
2e2f9fed55
rm duplicate imports
2019-12-11 11:11:56 -05:00
LysandreJik
4c12860f7a
Remove misleading documentation
2019-12-11 09:22:37 -05:00
Thomas Wolf
51ae203290
Merge pull request #2129 from leopd/master
...
Progress indicator improvements when downloading pre-trained models.
2019-12-10 22:18:55 +01:00
Leo Dirac
58d75aa310
Progress indicator improvements when downloading pre-trained models.
2019-12-10 11:36:56 -08:00
LysandreJik
6a73382706
Complete warning + cleanup
2019-12-10 14:33:24 -05:00
Lysandre
dc4e9e5cb3
DataParallel for SQuAD + fix XLM
2019-12-10 19:21:20 +00:00
Thomas Wolf
e6cff60b4c
Merge pull request #2069 from huggingface/cleaner-pt-tf-conversion
...
clean up PT <=> TF conversion
2019-12-10 15:34:08 +01:00
Rémi Louf
4b82c485de
remove misplaced summarization documentation
2019-12-10 09:13:33 -05:00
Thomas Wolf
e57d00ee10
Merge pull request #1984 from huggingface/squad-refactor
...
[WIP] Squad refactor
2019-12-10 11:07:26 +01:00
Thomas Wolf
ecabbf6d28
Merge pull request #2107 from huggingface/encoder-mask-shape
...
create encoder attention mask from shape of hidden states
2019-12-10 10:07:56 +01:00
Julien Chaumond
1d18930462
Harmonize no_cuda flag with other scripts
2019-12-09 20:37:55 -05:00
Rémi Louf
f7eba09007
clean for release
2019-12-09 20:37:55 -05:00
Rémi Louf
2a64107e44
improve device usage
2019-12-09 20:37:55 -05:00
Rémi Louf
c0707a85d2
add README
2019-12-09 20:37:55 -05:00
Rémi Louf
ade3cdf5ad
integrate ROUGE
2019-12-09 20:37:55 -05:00
Rémi Louf
076602bdc4
prevent BERT weights from being downloaded twice
2019-12-09 20:37:55 -05:00
Rémi Louf
5909f71028
add py-rouge dependency
2019-12-09 20:37:55 -05:00
Rémi Louf
a1994a71ee
simplified model and configuration
2019-12-09 20:37:55 -05:00
Rémi Louf
3a9a9f7861
default output dir to documents dir
2019-12-09 20:37:55 -05:00
Rémi Louf
693606a75c
update the docs
2019-12-09 20:37:55 -05:00
Rémi Louf
c0443df593
remove beam search
2019-12-09 20:37:55 -05:00
Rémi Louf
2403a66598
give transformers API to BertAbs
2019-12-09 20:37:55 -05:00
Rémi Louf
4d18199902
cast bool tensor to long for pytorch < 1.3
2019-12-09 20:37:55 -05:00
Rémi Louf
9f75565ea8
setup training
2019-12-09 20:37:55 -05:00
Rémi Louf
4735c2af07
tweaks to the BeamSearch API
2019-12-09 20:37:55 -05:00
Rémi Louf
ba089c780b
share pretrained embeddings
2019-12-09 20:37:55 -05:00
Rémi Louf
9660ba1cbd
Add beam search
2019-12-09 20:37:55 -05:00
Rémi Louf
1c71ecc880
load the pretrained weights for encoder-decoder
...
We currently save the pretrained_weights of the encoder and decoder in
two separate directories `encoder` and `decoder`. However, for the
`from_pretrained` function to operate with automodels we need to
specify the type of model in the path to the weights.
The path to the encoder/decoder weights is handled by the
`PreTrainedEncoderDecoder` class in the `save_pretrained` function. Sice
there is no easy way to infer the type of model that was initialized for
the encoder and decoder we add a parameter `model_type` to the function.
This is not an ideal solution as it is error prone, and the model type
should be carried by the Model classes somehow.
This is a temporary fix that should be changed before merging.
2019-12-09 20:37:55 -05:00
Rémi Louf
07f4cd73f6
update function to add special tokens
...
Since I started my PR the `add_special_token_single_sequence` function
has been deprecated for another; I replaced it with the new function.
2019-12-09 20:37:55 -05:00
Pierric Cistac
5c877fe94a
fix albert links
2019-12-09 18:53:00 -05:00
Bilal Khan
79526f82f5
Remove unnecessary epoch variable
2019-12-09 16:24:35 -05:00
Bilal Khan
9626e0458c
Add functionality to continue training from last saved global_step
2019-12-09 16:24:35 -05:00
Bilal Khan
2d73591a18
Stop saving current epoch
2019-12-09 16:24:35 -05:00
Bilal Khan
0eb973b0d9
Use saved optimizer and scheduler states if available
2019-12-09 16:24:35 -05:00
Bilal Khan
a03fcf570d
Save tokenizer after each epoch to be able to resume training from a checkpoint
2019-12-09 16:24:35 -05:00
Bilal Khan
f71b1bb05a
Save optimizer state, scheduler state and current epoch
2019-12-09 16:24:35 -05:00
LysandreJik
2a4ef098d6
Add ALBERT and XLM to SQuAD script
2019-12-09 10:46:47 -05:00
Julien Chaumond