c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,389 Commits 171 Branches 49 Tags
9660ba1cbdec0e419937af06bd99f06fb5ebbf91
Commit Graph
2389 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Rémi Louf 9660ba1cbd Add beam search 2019-12-09 20:37:55 -05:00
Rémi Louf 1c71ecc880 load the pretrained weights for encoder-decoder 
We currently save the pretrained_weights of the encoder and decoder in
two separate directories `encoder` and `decoder`. However, for the
`from_pretrained` function to operate with automodels we need to
specify the type of model in the path to the weights.

The path to the encoder/decoder weights is handled by the
`PreTrainedEncoderDecoder` class in the `save_pretrained` function. Sice
there is no easy way to infer the type of model that was initialized for
the encoder and decoder we add a parameter `model_type` to the function.
This is not an ideal solution as it is error prone, and the model type
should be carried by the Model classes somehow.

This is a temporary fix that should be changed before merging.
 2019-12-09 20:37:55 -05:00
Rémi Louf 07f4cd73f6 update function to add special tokens 
Since I started my PR the `add_special_token_single_sequence` function
has been deprecated for another; I replaced it with the new function.
 2019-12-09 20:37:55 -05:00
Pierric Cistac 5c877fe94a fix albert links 2019-12-09 18:53:00 -05:00
Bilal Khan 79526f82f5 Remove unnecessary epoch variable 2019-12-09 16:24:35 -05:00
Bilal Khan 9626e0458c Add functionality to continue training from last saved global_step 2019-12-09 16:24:35 -05:00
Bilal Khan 2d73591a18 Stop saving current epoch 2019-12-09 16:24:35 -05:00
Bilal Khan 0eb973b0d9 Use saved optimizer and scheduler states if available 2019-12-09 16:24:35 -05:00
Bilal Khan a03fcf570d Save tokenizer after each epoch to be able to resume training from a checkpoint 2019-12-09 16:24:35 -05:00
Bilal Khan f71b1bb05a Save optimizer state, scheduler state and current epoch 2019-12-09 16:24:35 -05:00
Aymeric Augustin 0cb163865a Remove pytest dependency. (#2093) 2019-12-07 07:46:14 -05:00
Michael Watkins 2670b0d682 Fix bug which lowercases special tokens 2019-12-06 16:15:53 -05:00
Aymeric Augustin 35401fe50f Remove dependency on pytest for running tests (#2055) 
* Switch to plain unittest for skipping slow tests.

Add a RUN_SLOW environment variable for running them.

* Switch to plain unittest for PyTorch dependency.

* Switch to plain unittest for TensorFlow dependency.

* Avoid leaking open files in the test suite.

This prevents spurious warnings when running tests.

* Fix unicode warning on Python 2 when running tests.

The warning was:

    UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal

* Support running PyTorch tests on a GPU.

Reverts 27e015bd.

* Tests no longer require pytest.

* Make tests pass on cuda
 2019-12-06 13:57:38 -05:00
Julien Chaumond e4679cddce [cli] Uploads: add progress bar (#2078) 
* [cli] Uploads: add progress bar

see https://github.com/huggingface/transformers/pull/2044#discussion_r354057827 for context

* rename + documentation

* Add auto-referential comment
 2019-12-06 11:56:23 -05:00
Thomas Wolf 4cb9b60558 Merge pull request #2077 from patrickvonplaten/change_documentation_for_past_output_shape 
corrected documentation for past tensor shape for ctrl and gpt2 model
 2019-12-06 12:14:48 +01:00
Thomas Wolf 5482822a2b Merge pull request #2046 from jplu/tf2-ner-example 
Add NER TF2 example.
 2019-12-06 12:12:22 +01:00
Thomas Wolf fc1bb1f867 Merge pull request #2068 from huggingface/fix-2042 
Nicer error message when Bert's input is missing batch size
 2019-12-06 12:06:42 +01:00
patrickvonplaten d0383e4daf corrected documentation for past tensor shape for ctrl and gpt2 model 2019-12-06 01:24:22 +01:00
VictorSanh 35ff345fc9 update requirements 2019-12-05 12:07:04 -05:00
VictorSanh 552c44a9b1 release distilm-bert 2019-12-05 10:14:58 -05:00
Rosanne Liu ee53de7aac Pr for pplm (#2060) 
* license

* changes

* ok

* Update paper link and commands to run

* pointer to uber repo
 2019-12-05 09:20:07 -05:00
Thomas Wolf bebaa14039 Merge pull request #2045 from aaugustin/remove-dead-code 
Remove dead code in tests.
 2019-12-05 14:41:56 +01:00
thomwolf 18fb93530b fixing #2042 - Nicer error message 2019-12-05 14:36:34 +01:00
thomwolf 2d5d86e037 fix #2031 2019-12-05 14:06:29 +01:00
Thomas Wolf af077b15e2 Merge pull request #2065 from huggingface/fixing-camembert 
Fixing camembert tokenization
 2019-12-05 13:45:44 +01:00
thomwolf 3268ebd229 fix xlnet test 2019-12-05 13:35:29 +01:00
thomwolf 6c5297a423 Fixing camembert tokenization 2019-12-05 13:27:58 +01:00
Julien Plu 9200a759d7 Add few tests on the TF optimization file with some info in the documentation. Complete the README. 2019-12-05 12:56:43 +01:00
Thomas Wolf 1f179f095f Merge pull request #2011 from AdityaSoni19031997/patch-1 
typo fix on the docs as per Pytorch v1.1+
 2019-12-05 12:39:04 +01:00
Thomas Wolf 1eaf44e713 Merge pull request #2007 from roskoN/xlnet_attention_fix 
fixed XLNet attention output for both attention streams whenever target_mapping is provided
 2019-12-05 12:32:39 +01:00
thomwolf 71e4693f08 fix #1968 2019-12-05 12:14:24 +01:00
Thomas Wolf f9f395b21c Merge pull request #1735 from ondewo/tf-do-not-use-gpu-on-import 
Do not use GPU when importing transformers
 2019-12-05 11:56:48 +01:00
thomwolf 75a97af6bc fix #1450 - add doc 2019-12-05 11:26:55 +01:00
thomwolf 8b388827b5 fix #1920 2019-12-05 11:18:43 +01:00
Thomas Wolf d425a4d60b Merge pull request #1870 from alexzubiaga/xlnet-for-token-classification 
XLNet for Token classification
 2019-12-05 09:54:09 +01:00
Thomas Wolf 1eb89ddf73 Merge pull request #2044 from huggingface/cli_upload 
CLI for authenticated file sharing
 2019-12-05 09:44:07 +01:00
VictorSanh fb0d2f1da1 preparing release distil-mBERT 2019-12-05 03:00:16 -05:00
Julien Chaumond 3ba417e1a8 [cli] ls: Tabular formatting 2019-12-04 18:40:52 -05:00
Julien Chaumond 96fa9a8a70 Python 2 + Post mime-type to S3 2019-12-04 17:22:50 -05:00
Julien Plu ff98b041da Fix whitespace issue 2019-12-04 16:53:06 +01:00
thomwolf 5bfcd0485e fix #1991 2019-12-04 14:53:11 +01:00
Thomas Wolf cae641ff26 Merge pull request #1846 from tamuhey/patch/iss1845 
fix summary_type value of SequenceSummary
 2019-12-04 13:28:39 +01:00
Julien Plu 254ebb979c Bugfix on init file. Missing comma. 2019-12-04 10:00:25 +01:00
Julien Plu ecb923da9c Create a NER example similar to the Pytorch one. It takes the same options, and can be run the same way. 2019-12-04 09:43:15 +01:00
Aymeric Augustin 40255ab002 Remove dead code in tests. 2019-12-04 08:21:02 +01:00
Julien Chaumond e4fbf3e2cc CLI for authenticated file sharing 2019-12-04 00:52:23 -05:00
Julien Chaumond 7edb51f3a5 [pplm] split classif head into its own file 2019-12-03 22:07:25 +00:00
LysandreJik 8101924a68 Patch: v2.2.1 v2.2.1 2019-12-03 11:20:26 -05:00
VictorSanh 48cbf267c9 Use full dataset for eval (SequentialSampler in Distributed setting) 2019-12-03 11:01:37 -05:00
Julien ChaumondandPiero Molino f434bfc623 [pplm] Update S3 links 
Co-Authored-By: Piero Molino <w4nderlust@gmail.com>
 2019-12-03 10:53:02 -05:00