c-my/transformers
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,917 Commits 171 Branches 49 Tags
4bb01a437348ec822f6bc5a5c4a574be18f1e881
Commit Graph
2917 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
thomwolf 4bb01a4373 code formating 2019-12-28 17:35:10 +01:00
thomwolf 100bbcb786 GPT2 should not store/compute cached activations during finetuning 2019-12-28 17:24:52 +01:00
Thomas Wolf f75bf05ce6 Merge pull request #2352 from huggingface/cli_tweaks 
Cli tweaks
 2019-12-28 15:40:00 +01:00
Julien Chaumond 0d467fd6de Typo 2019-12-27 23:06:48 -05:00
Julien Chaumond d8293e84f3 [cli] upload: max number of files at the same time 2019-12-27 23:02:53 -05:00
Julien Chaumond 4d6c93e923 Kill __main__ 2019-12-27 22:55:22 -05:00
Julien Chaumond 9b2badf3c9 [cli] Update doc 2019-12-27 22:54:29 -05:00
Julien Chaumond f78ebc22ad [cli] Add ability to delete remote object 2019-12-27 22:53:49 -05:00
Anthony MOI bfe870be65 Hotfix tokenizers version for sdist installs 2019-12-27 11:05:52 -05:00
Thomas Wolf 74ea432847 Merge pull request #2286 from adelevie/patch-2 
Typo in tokenization_utils.py
 2019-12-27 10:50:47 +01:00
Thomas Wolf 492bea9aa0 Merge pull request #2292 from patrickvonplaten/add_cached_past_for_language_generation 
Add cached past for language generation
 2019-12-27 10:33:27 +01:00
Thomas Wolf e213900fa2 Merge pull request #2290 from patrickvonplaten/fix_typo_in_doc_for_language_generation 
duplicated line for repeating_words_penalty_for_language_generation
 2019-12-27 10:29:06 +01:00
Thomas Wolf 9f5f646442 Merge pull request #2211 from huggingface/fast-tokenizers 
Fast tokenizers
 2019-12-27 10:24:29 +01:00
Aymeric Augustin 9024b19994 Auto-format (fixes previous commit). 2019-12-27 10:13:52 +01:00
Aymeric Augustin 3233b58ad4 Quote square brackets in shell commands. 
This ensures compatibility with zsh.

Fix #2316.
 2019-12-27 08:50:25 +01:00
Anthony MOI e6ec24fa88 Better added_tokens handling 2019-12-26 16:49:48 -05:00
Anthony MOI 599db139f9 Code style update 2019-12-26 15:13:30 -05:00
Anthony MOI 835b76a46f Handle unk_token 
As we discussed, this is handled here directly 
cc @thomwolf
 2019-12-26 14:42:55 -05:00
Anthony MOI 7ead04ce14 FastPreTrainedTokenizer => PreTrainedTokenizerFast 2019-12-26 14:39:39 -05:00
Anthony MOI 1f82a5d910 Update for changes in tokenizers API 2019-12-26 14:37:55 -05:00
Thomas Wolf 8c67b529f6 Merge pull request #2324 from kashif/patch-1 
Typo in serving.py
 2019-12-26 12:38:06 +01:00
Kashif Rasul 7211541ade Typo in serving.py 2019-12-26 12:21:40 +01:00
patrickvonplaten 0f6017bee3 improve comments for examples 2019-12-26 00:35:11 +01:00
patrickvonplaten 87c8fca9bc add example for ctrl text generation in docs 2019-12-26 00:29:19 +01:00
patrickvonplaten 88def24c45 merge conflicts - renamed to previous_token singular 2019-12-26 00:27:16 +01:00
patrickvonplaten 822f725a07 duplicated line for repeating_words_penalty_for_language_generation 2019-12-26 00:25:29 +01:00
patrickvonplaten fc84bd5254 adapt style to predefined style layout 2019-12-25 23:32:44 +01:00
patrickvonplaten deff792bb6 add prepare inputs for transfo_xl and xlnet 2019-12-25 23:17:24 +01:00
patrickvonplaten 9398058e19 add easy tensor shape match test 2019-12-25 23:17:24 +01:00
patrickvonplaten 90cda45e9e add past re-ordering for beam search 2019-12-25 23:17:24 +01:00
patrickvonplaten 6bca56fdb0 check for self.config.mem_len instead of self.mem_len in _do_output_past 2019-12-25 23:17:24 +01:00
patrickvonplaten 365ccd0af2 make if statements cleaner for prepare_inputs_for_generation 2019-12-25 23:17:24 +01:00
patrickvonplaten d039c679d2 better naming for if statement 2019-12-25 23:17:24 +01:00
patrickvonplaten 7e0c5c731a changed do_output_past function to check for self.config.output_past instead of self.output_past 2019-12-25 23:17:24 +01:00
patrickvonplaten eeaa402cd4 rename comments 2019-12-25 23:17:24 +01:00
patrickvonplaten 7bb4271291 remove ipdb debugging statements 2019-12-25 23:17:24 +01:00
patrickvonplaten 267587c258 add and improve comments 2019-12-25 23:17:24 +01:00
patrickvonplaten d891fd0ae0 add past hidden key states for more efficient language generation & add prepare_inputs for gpt2 and ctrl model 2019-12-25 23:17:24 +01:00
Thomas Wolf aeef4823ab Merge pull request #2303 from patrickvonplaten/fix_error_with_repetition_penalty 
fix repetition penalty error in modeling_utils.py
 2019-12-25 22:39:20 +01:00
Thomas Wolf 0412f3d929 Merge pull request #2291 from aaugustin/fix-flake8-F841 
Fix F841 flake8 warning
 2019-12-25 22:37:42 +01:00
Thomas Wolf 8742c95461 Merge pull request #2289 from patrickvonplaten/fix_effective_batch_size_lang_gen_xlm 
fix bug in prepare inputs for language generation for xlm for effective batch_size > 1
 2019-12-25 22:30:46 +01:00
Thomas Wolf 1240be3ed9 Merge pull request #2312 from vitaliyradchenko/fix_special_and_add_tokens_loading 
Correct tokenization for special and added tokens
 2019-12-25 20:52:30 +01:00
vitaliyradchenko b262577d17 add special tokens to unique_added_tokens_encoder 2019-12-25 18:31:35 +02:00
vitaliyradchenko 83a2347952 fixed lack of added and special tokens 2019-12-25 18:03:19 +02:00
Thomas Wolf cea04a2443 Merge pull request #2310 from ShnitzelKiller/scatter-unfix 
revert erroneous fix #2276
 2019-12-25 12:43:22 +01:00
James Noeckel e1844d9a45 use positional arguments due to inconsistent API 2019-12-25 01:34:02 -08:00
James Noeckel 9fb7addd4d revert erroneous fix 2019-12-24 22:26:09 -08:00
Anthony MOI 734d29b03d tokenizers is now a real dependency 2019-12-24 13:32:41 -05:00
Anthony MOI 2818e50569 Add tests for fast tokenizers 2019-12-24 13:29:01 -05:00
Anthony MOI 31c56f2e0b Fix style 2019-12-24 12:43:27 -05:00