c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
906 Commits 171 Branches 49 Tags
68ab9599ce3aefbd25d1c81e3315d1968849b628
Commit Graph
20 Commits
Author SHA1 Message Date
thomwolf 8415a38b23 better error messages 2019-06-17 13:03:48 +02:00
Thomas Wolf ff276fc00c Merge branch 'master' into finish_torchhub_interfaces 2019-06-14 16:59:07 +02:00
VictorSanh a92b6dc3c1 add GPT2 torchhub compatibility 2019-06-01 15:27:43 -04:00
thomwolf 366a3b0285 clean up in tokenization 2019-05-08 21:43:51 +02:00
thomwolf d1b6979aa5 GPT-2 option to avoid predicting special tokens 2019-05-07 16:25:53 +02:00
thomwolf db98a4a48b gpt-2 tokenizer 2019-05-01 11:40:48 +02:00
Ben Mann 74f7906db4 Fix #537 2019-04-30 19:48:22 -07:00
thomwolf 929579f3b5 fix #497 2019-04-17 12:35:08 +02:00
thomwolf 5afa497cbf fix GPT-2 tokenization to work also on python 3... 2019-04-17 11:04:41 +02:00
thomwolf bc70779bf0 fixed GPT-2 tokenization on python 2 2019-04-17 10:56:15 +02:00
thomwolf 18a8a15f78 improving GPT2 tokenization and adding tests 2019-04-16 17:00:55 +02:00
thomwolf e8568a3b17 fixing tests 2019-04-15 12:55:38 +02:00
thomwolf 870b734bfd added tokenizers serialization tests 2019-04-15 12:03:56 +02:00
thomwolf 3e65f255dc add serialization semantics to tokenizers - fix transfo-xl tokenizer 2019-04-15 11:47:25 +02:00
thomwolf 5c85fc3977 fix typo - logger info 2019-03-06 10:05:21 +01:00
Catalin Voss 4a49c22584 Warn instead of raising in BERT and GPT-2 tokenizers as well, to allow for pre-caching of tokens 2019-03-05 12:31:45 -08:00
thomwolf ab7f5d2943 simple 2019-02-18 11:33:54 +01:00
thomwolf b450a7faf2 clean up tokenization - fix python 2 tests 2019-02-18 11:27:18 +01:00
thomwolf d44db1145c update readme 2019-02-18 11:12:09 +01:00
thomwolf ffd623823d adding gpt2 2019-02-17 23:38:51 +01:00