c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,569 Commits 171 Branches 49 Tags
f7978f70ecadd3ac6e3072f72e9dcc8c01e187c8
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
thomwolf f7978f70ec use format instead of f-strings 2019-10-01 18:45:38 -04:00
thomwolf e768f2322a update run_openai_gpt to fix #1264 2019-09-18 10:07:47 +02:00
thomwolf 8334993915 clean up examples - updated to new keyword inputs - #1246 2019-09-18 10:01:27 +02:00
thomwolf 45de034bf8 fix #1223 2019-09-17 10:25:06 +02:00
thomwolf c88f05163d fix typo in XLM models 2019-09-16 13:42:20 +02:00
thomwolf 3401980fc4 fix #1208 2019-09-09 10:22:12 +03:00
thomwolf ee027c89f2 fix #1165 2019-09-06 23:40:05 +03:00
thomwolf 5c6cac102b adding test for common properties and cleaning up a bit base class 2019-09-05 21:31:29 +02:00
thomwolf 85df4f7cca also gathering file names in file_utils 2019-09-05 02:34:09 +02:00
thomwolf 121f88cae3 update conversion scripts 2019-09-05 02:17:50 +02:00
thomwolf d77abd4d08 clean ups 2019-09-05 00:41:24 +02:00
thomwolf 2a667b1eb9 split configuration and modeling files 2019-09-05 00:27:11 +02:00
thomwolf 0be6a2a624 be sure we have uint8 2019-09-04 22:47:38 +02:00
thomwolf 7fba47b7d9 WIP reordering 2019-09-04 22:39:23 +02:00
thomwolf e25cba78cf WIP reodering arguments for torchscript and TF 2019-09-04 22:39:23 +02:00
thomwolf 38b79b5a63 Fixing this TransformerXL bool issue 2019-09-04 22:36:30 +02:00
thomwolf 89fd3450a6 Release: 1.2.0 2019-09-04 13:32:18 +02:00
thomwolf fede4ef45d fixing #1133 2019-09-02 02:27:39 +02:00
thomwolf bdb4409ed8 updated pruning logic with sets - Bert and GPT-2 2019-08-31 00:33:50 -04:00
thomwolf bfe93a5a21 fix distilbert in auto tokenizer 2019-08-30 22:43:26 +02:00
thomwolf 256086bc69 clean up and simplify hubconf 2019-08-30 22:34:23 +02:00
thomwolf 80aa87d9a3 fix distilbert tokenizer 2019-08-30 22:24:23 +02:00
thomwolf 455a4c842c add distilbert tokenizer 2019-08-30 22:20:51 +02:00
thomwolf 7a1f174a9d update names of torch.hub to simpler names - update docstring 2019-08-30 22:20:44 +02:00
thomwolf c665e0fcfe Merge branch 'automodels' of https://github.com/huggingface/pytorch-transformers into automodels 2019-08-30 21:53:36 +02:00
thomwolf f35f612280 updating docstring for AutoModel 2019-08-30 12:48:55 -04:00
thomwolf 7044ed6b05 fix tokenizers serialization 2019-08-30 17:36:11 +02:00
thomwolf 69da972ace added test and debug tokenizer configuration serialization 2019-08-30 17:09:36 +02:00
thomwolf 88111de07c saving and reloading tokenizer configurations 2019-08-30 16:55:48 +02:00
thomwolf 3871b8a107 adding xlm 17 and 100 models and config on aws 2019-08-30 16:28:42 +02:00
thomwolf 8678ff8df5 adding 17 and 100 xlm models 2019-08-30 16:26:04 +02:00
thomwolf 82462c5cba Added option to setup pretrained tokenizer arguments 2019-08-30 15:30:41 +02:00
thomwolf f8aace6bcd update tokenizers to use self.XX_token_id instead of converting self.XX_token 2019-08-30 13:39:52 +02:00
thomwolf 8faf2e086b more doc on special tokens 2019-08-30 13:36:22 +02:00
thomwolf ce5ef4b35d python2 doesn't spark joy 2019-08-30 13:22:43 +02:00
thomwolf 5dd7b677ad clean up all byte-level bpe tests 2019-08-30 12:43:08 +02:00
thomwolf ca1a00a302 fix for python2 2019-08-30 12:29:31 +02:00
thomwolf 4e6a3172ce update roberta docstring as well 2019-08-30 12:23:37 +02:00
thomwolf fd10d79b55 update GPT2 docstring 2019-08-30 12:23:12 +02:00
thomwolf abe734ca1f fix GPT-2 and RoBERTa tests to be clean now 2019-08-30 12:20:18 +02:00
thomwolf 0f5a799456 fix GPT2DoubleHeadModel docstring 2019-08-30 11:49:23 +02:00
thomwolf d51f72d5de adding shortcut to the ids of all the special tokens 2019-08-30 11:41:11 +02:00
thomwolf 306af132d7 update readme to mention add_special_tokens more clearly in example 2019-08-30 11:30:51 +02:00
thomwolf 50e6daf83a fix Roberta tokenizer __init__ 2019-08-30 11:27:43 +02:00
thomwolf 0517e7a1cb Fix GPT2 and RoBERTa tokenizer to beging with a space - update Roberta tokenizer 2019-08-30 11:23:49 +02:00
thomwolf 0a74c88ac6 fix #1131 2019-08-28 22:41:42 +02:00
thomwolf e7706f514b update again 2019-08-28 16:37:22 +02:00
thomwolf b5eb283aaa update credits 2019-08-28 16:36:55 +02:00
thomwolf 912a377e90 dilbert -> distilbert 2019-08-28 13:59:42 +02:00
thomwolf c9bce1811c fixing model to add torchscript, embedding resizing, head pruning and masking + tests 2019-08-28 13:22:45 +02:00
thomwolf 62df4ba59a add dilbert tokenizer and tests 2019-08-28 12:22:56 +02:00
thomwolf 4ce5f36f78 update readmes 2019-08-28 12:14:31 +02:00
thomwolf 0d288727b8 fix #1106 2019-08-27 14:50:22 +02:00
thomwolf 447afe9cdf updating docstring for AutoModel 2019-08-27 14:42:03 +02:00
thomwolf a175a9dc01 add kwargs to base encode function 2019-08-27 14:05:59 +02:00
thomwolf 06510ccb53 typo 2019-08-23 22:08:10 +02:00
thomwolf 3bcbebd440 max_len_single_sentence & max_len_sentences_pair as attributes so they can be modified 2019-08-23 22:07:26 +02:00
thomwolf ab7bd5ef98 fixing tokenization and training 2019-08-23 17:31:21 +02:00
thomwolf 47d6853439 adding max_lengths for single sentences and sentences pairs 2019-08-23 17:31:11 +02:00
thomwolf e00b4ff1de fix #1017 2019-08-21 22:22:17 +02:00
thomwolf fdc487d8b3 Add max length 2019-08-21 02:35:01 +02:00
thomwolf aa05dc8935 adding gpt-2 large 2019-08-21 02:29:34 +02:00
thomwolf 43489756ad adding proxies options for the from_pretrained methods 2019-08-20 16:59:11 +02:00
thomwolf a690edab17 various fix and clean up on run_lm_finetuning 2019-08-20 15:52:12 +02:00
thomwolf 6d0aa73981 fix #1034 2019-08-20 12:20:21 +02:00
thomwolf 53c8f700f4 fix #808 2019-08-20 11:29:26 +02:00
thomwolf 901dde0e45 fix #1014 2019-08-20 11:05:51 +02:00
thomwolf e239a4a20f close #984 2019-08-20 11:02:00 +02:00
thomwolf fecaed0ed4 add force_download option to from_pretrained methods 2019-08-20 10:56:12 +02:00
thomwolf aaedfc35a8 Merge branch 'master' of https://github.com/huggingface/pytorch-transformers 2019-08-10 20:04:37 +02:00
thomwolf c683c3d5a5 fix #993 2019-08-10 20:04:35 +02:00
thomwolf 0b524b0848 remove derived classes for now 2019-08-05 19:08:19 +02:00
thomwolf 13936a9621 update doc and tests 2019-08-05 18:48:16 +02:00
thomwolf ed4e542260 adding tests 2019-08-05 18:14:07 +02:00
thomwolf 3a126e73dd fix #950 2019-08-05 17:26:29 +02:00
thomwolf 7223886dc9 fix #944 2019-08-05 17:16:56 +02:00
thomwolf 70c10caa06 add option mentioned in #940 2019-08-05 17:09:37 +02:00
thomwolf 077ad693e9 tweak issue templates wordings 2019-08-05 16:46:29 +02:00
thomwolf 02d4087cb8 Merge branch 'master' of https://github.com/huggingface/pytorch-pretrained-BERT 2019-08-05 16:26:01 +02:00
thomwolf 7c524d631e add issue templates 2019-08-05 16:25:54 +02:00
thomwolf b90e29d52c working on automodels 2019-08-05 16:06:34 +02:00
thomwolf 58830807d1 inidicate we only support pytorch 1.0.0+ now 2019-08-05 14:38:59 +02:00
thomwolf 328afb7097 cleaning up tokenizer tests structure (at last) - last remaining ppb refs 2019-08-05 14:08:56 +02:00
thomwolf 00132b7a7a updating docs - adding few tests to tokenizers 2019-08-04 22:42:55 +02:00
thomwolf 009273dbdd big doc update [WIP] 2019-08-04 12:14:57 +02:00
thomwolf bfbe52ec39 cleaning up example docstrings 2019-07-27 20:25:39 +02:00
thomwolf 4cc1bf81ee typos 2019-07-27 12:08:21 +02:00
thomwolf ac27548b25 fix unk_token test 2019-07-27 11:50:47 +02:00
thomwolf c717d38573 dictionnary => dictionary 2019-07-26 23:30:48 +02:00
thomwolf 7b6e474c9a fix #901 2019-07-26 21:26:44 +02:00
thomwolf 632d711411 fix #908 2019-07-26 21:14:37 +02:00
thomwolf 27b0f86d36 clean up pretrained 2019-07-26 17:09:21 +02:00
thomwolf 57e54ec070 add unk_token to gpt2 2019-07-26 17:09:07 +02:00
thomwolf ac42049c08 add auto models and auto tokenizer 2019-07-26 17:08:59 +02:00
thomwolf 1383c7b87a Fix #869 2019-07-23 17:52:20 +02:00
thomwolf 6070b55443 fix #868 2019-07-23 17:46:01 +02:00
thomwolf 2c9a3115b7 fix #858 2019-07-23 16:45:55 +02:00
thomwolf 0740e63e49 updating schedules for state_dict saving 2019-07-23 15:57:18 +02:00
thomwolf ba52fe69d5 update breaking change section regarding from_pretrained keyword arguments 2019-07-23 15:10:02 +02:00
thomwolf 0227b4a940 fix #827 2019-07-23 14:06:43 +02:00