c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,462 Commits 171 Branches 49 Tags
a1c34bd286d57625b844703705a39e40737a21ba
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
thomwolf bfe93a5a21 fix distilbert in auto tokenizer 2019-08-30 22:43:26 +02:00
thomwolf 256086bc69 clean up and simplify hubconf 2019-08-30 22:34:23 +02:00
thomwolf 80aa87d9a3 fix distilbert tokenizer 2019-08-30 22:24:23 +02:00
thomwolf 455a4c842c add distilbert tokenizer 2019-08-30 22:20:51 +02:00
thomwolf 7a1f174a9d update names of torch.hub to simpler names - update docstring 2019-08-30 22:20:44 +02:00
thomwolf c665e0fcfe Merge branch 'automodels' of https://github.com/huggingface/pytorch-transformers into automodels 2019-08-30 21:53:36 +02:00
thomwolf f35f612280 updating docstring for AutoModel 2019-08-30 12:48:55 -04:00
thomwolf 7044ed6b05 fix tokenizers serialization 2019-08-30 17:36:11 +02:00
thomwolf 69da972ace added test and debug tokenizer configuration serialization 2019-08-30 17:09:36 +02:00
thomwolf 88111de07c saving and reloading tokenizer configurations 2019-08-30 16:55:48 +02:00
thomwolf 3871b8a107 adding xlm 17 and 100 models and config on aws 2019-08-30 16:28:42 +02:00
thomwolf 8678ff8df5 adding 17 and 100 xlm models 2019-08-30 16:26:04 +02:00
thomwolf 82462c5cba Added option to setup pretrained tokenizer arguments 2019-08-30 15:30:41 +02:00
thomwolf f8aace6bcd update tokenizers to use self.XX_token_id instead of converting self.XX_token 2019-08-30 13:39:52 +02:00
thomwolf 8faf2e086b more doc on special tokens 2019-08-30 13:36:22 +02:00
thomwolf ce5ef4b35d python2 doesn't spark joy 2019-08-30 13:22:43 +02:00
thomwolf 5dd7b677ad clean up all byte-level bpe tests 2019-08-30 12:43:08 +02:00
thomwolf ca1a00a302 fix for python2 2019-08-30 12:29:31 +02:00
thomwolf 4e6a3172ce update roberta docstring as well 2019-08-30 12:23:37 +02:00
thomwolf fd10d79b55 update GPT2 docstring 2019-08-30 12:23:12 +02:00
thomwolf abe734ca1f fix GPT-2 and RoBERTa tests to be clean now 2019-08-30 12:20:18 +02:00
thomwolf 0f5a799456 fix GPT2DoubleHeadModel docstring 2019-08-30 11:49:23 +02:00
thomwolf d51f72d5de adding shortcut to the ids of all the special tokens 2019-08-30 11:41:11 +02:00
thomwolf 306af132d7 update readme to mention add_special_tokens more clearly in example 2019-08-30 11:30:51 +02:00
thomwolf 50e6daf83a fix Roberta tokenizer __init__ 2019-08-30 11:27:43 +02:00
thomwolf 0517e7a1cb Fix GPT2 and RoBERTa tokenizer to beging with a space - update Roberta tokenizer 2019-08-30 11:23:49 +02:00
thomwolf 0a74c88ac6 fix #1131 2019-08-28 22:41:42 +02:00
thomwolf e7706f514b update again 2019-08-28 16:37:22 +02:00
thomwolf b5eb283aaa update credits 2019-08-28 16:36:55 +02:00
thomwolf 912a377e90 dilbert -> distilbert 2019-08-28 13:59:42 +02:00
thomwolf c9bce1811c fixing model to add torchscript, embedding resizing, head pruning and masking + tests 2019-08-28 13:22:45 +02:00
thomwolf 62df4ba59a add dilbert tokenizer and tests 2019-08-28 12:22:56 +02:00
thomwolf 4ce5f36f78 update readmes 2019-08-28 12:14:31 +02:00
thomwolf 0d288727b8 fix #1106 2019-08-27 14:50:22 +02:00
thomwolf 447afe9cdf updating docstring for AutoModel 2019-08-27 14:42:03 +02:00
thomwolf a175a9dc01 add kwargs to base encode function 2019-08-27 14:05:59 +02:00
thomwolf 06510ccb53 typo 2019-08-23 22:08:10 +02:00
thomwolf 3bcbebd440 max_len_single_sentence & max_len_sentences_pair as attributes so they can be modified 2019-08-23 22:07:26 +02:00
thomwolf ab7bd5ef98 fixing tokenization and training 2019-08-23 17:31:21 +02:00
thomwolf 47d6853439 adding max_lengths for single sentences and sentences pairs 2019-08-23 17:31:11 +02:00
thomwolf e00b4ff1de fix #1017 2019-08-21 22:22:17 +02:00
thomwolf fdc487d8b3 Add max length 2019-08-21 02:35:01 +02:00
thomwolf aa05dc8935 adding gpt-2 large 2019-08-21 02:29:34 +02:00
thomwolf 43489756ad adding proxies options for the from_pretrained methods 2019-08-20 16:59:11 +02:00
thomwolf a690edab17 various fix and clean up on run_lm_finetuning 2019-08-20 15:52:12 +02:00
thomwolf 6d0aa73981 fix #1034 2019-08-20 12:20:21 +02:00
thomwolf 53c8f700f4 fix #808 2019-08-20 11:29:26 +02:00
thomwolf 901dde0e45 fix #1014 2019-08-20 11:05:51 +02:00
thomwolf e239a4a20f close #984 2019-08-20 11:02:00 +02:00
thomwolf fecaed0ed4 add force_download option to from_pretrained methods 2019-08-20 10:56:12 +02:00
thomwolf aaedfc35a8 Merge branch 'master' of https://github.com/huggingface/pytorch-transformers 2019-08-10 20:04:37 +02:00
thomwolf c683c3d5a5 fix #993 2019-08-10 20:04:35 +02:00
thomwolf 0b524b0848 remove derived classes for now 2019-08-05 19:08:19 +02:00
thomwolf 13936a9621 update doc and tests 2019-08-05 18:48:16 +02:00
thomwolf ed4e542260 adding tests 2019-08-05 18:14:07 +02:00
thomwolf 3a126e73dd fix #950 2019-08-05 17:26:29 +02:00
thomwolf 7223886dc9 fix #944 2019-08-05 17:16:56 +02:00
thomwolf 70c10caa06 add option mentioned in #940 2019-08-05 17:09:37 +02:00
thomwolf 077ad693e9 tweak issue templates wordings 2019-08-05 16:46:29 +02:00
thomwolf 02d4087cb8 Merge branch 'master' of https://github.com/huggingface/pytorch-pretrained-BERT 2019-08-05 16:26:01 +02:00
thomwolf 7c524d631e add issue templates 2019-08-05 16:25:54 +02:00
thomwolf b90e29d52c working on automodels 2019-08-05 16:06:34 +02:00
thomwolf 58830807d1 inidicate we only support pytorch 1.0.0+ now 2019-08-05 14:38:59 +02:00
thomwolf 328afb7097 cleaning up tokenizer tests structure (at last) - last remaining ppb refs 2019-08-05 14:08:56 +02:00
thomwolf 00132b7a7a updating docs - adding few tests to tokenizers 2019-08-04 22:42:55 +02:00
thomwolf 009273dbdd big doc update [WIP] 2019-08-04 12:14:57 +02:00
thomwolf bfbe52ec39 cleaning up example docstrings 2019-07-27 20:25:39 +02:00
thomwolf 4cc1bf81ee typos 2019-07-27 12:08:21 +02:00
thomwolf ac27548b25 fix unk_token test 2019-07-27 11:50:47 +02:00
thomwolf c717d38573 dictionnary => dictionary 2019-07-26 23:30:48 +02:00
thomwolf 7b6e474c9a fix #901 2019-07-26 21:26:44 +02:00
thomwolf 632d711411 fix #908 2019-07-26 21:14:37 +02:00
thomwolf 27b0f86d36 clean up pretrained 2019-07-26 17:09:21 +02:00
thomwolf 57e54ec070 add unk_token to gpt2 2019-07-26 17:09:07 +02:00
thomwolf ac42049c08 add auto models and auto tokenizer 2019-07-26 17:08:59 +02:00
thomwolf 1383c7b87a Fix #869 2019-07-23 17:52:20 +02:00
thomwolf 6070b55443 fix #868 2019-07-23 17:46:01 +02:00
thomwolf 2c9a3115b7 fix #858 2019-07-23 16:45:55 +02:00
thomwolf 0740e63e49 updating schedules for state_dict saving 2019-07-23 15:57:18 +02:00
thomwolf ba52fe69d5 update breaking change section regarding from_pretrained keyword arguments 2019-07-23 15:10:02 +02:00
thomwolf 0227b4a940 fix #827 2019-07-23 14:06:43 +02:00
thomwolf 71d597dad0 fix #800 2019-07-17 13:51:09 +02:00
thomwolf 5fe0b378d8 adding missing docstring fix #793 2019-07-16 21:35:53 +02:00
thomwolf e848b54730 fix #792 2019-07-16 21:22:19 +02:00
thomwolf c5b3d86a91 Merge branch 'master' of https://github.com/huggingface/pytorch-pretrained-BERT 2019-07-16 21:21:05 +02:00
thomwolf 6b70760204 typos 2019-07-16 21:21:03 +02:00
thomwolf ed7549bb1a release version 1.0 2019-07-16 16:10:58 +02:00
thomwolf 6a72d9aa52 updated examples in readme 2019-07-16 16:09:29 +02:00
thomwolf b59043bf8f update readme 2019-07-16 16:03:48 +02:00
thomwolf edc79acb3b simpler quick tour 2019-07-16 16:02:32 +02:00
thomwolf 5c82d3488f indicate default evaluation in breaking changes 2019-07-16 15:45:58 +02:00
thomwolf 4acaa65068 model in evaluation mode by default after from_pretrained 2019-07-16 15:41:57 +02:00
thomwolf f289e6cfe4 fix docstrings 2019-07-16 15:31:21 +02:00
thomwolf 9726b229cf model name typo 2019-07-16 15:17:45 +02:00
thomwolf 1849aa7d39 update readme and pretrained model weight files 2019-07-16 15:11:29 +02:00
thomwolf 43e0e8fa04 updates to readme and doc 2019-07-16 13:56:47 +02:00
thomwolf f31154cb9d Merge branch 'xlnet' 2019-07-16 11:51:13 +02:00
thomwolf 1b35d05d4b update conversion scripts and __main__ 2019-07-16 09:41:55 +02:00
thomwolf 352e3ff998 added migration guide to readme 2019-07-16 09:03:49 +02:00
thomwolf 8ad7e5b4f2 indeed 2019-07-16 00:29:15 +02:00