6 Commits

Include renames

Author	SHA1	Message	Date
thomwolf	15d8b1266c	update tokenizer - update squad example for xlnet	2019-07-15 17:30:42 +02:00
thomwolf	699bc7e86e	fix gpt-2 unk token test	2019-07-12 11:46:57 +02:00
LysandreJik	e3fb4310d6	From pretrained correct initialization. Unknown token handling for gpt2.	2019-07-11 18:44:29 -04:00
thomwolf	b19786985d	unified tokenizer api and serialization + tests	2019-07-09 10:25:18 +02:00
thomwolf	36bca545ff	tokenization abstract class - tests for examples	2019-07-05 15:02:59 +02:00
thomwolf	0bab55d5d5	[BIG] name change	2019-07-05 11:55:36 +02:00