8 Commits

Include renames

Author	SHA1	Message	Date
thomwolf	3bcbebd440	max_len_single_sentence & max_len_sentences_pair as attributes so they can be modified	2019-08-23 22:07:26 +02:00
Guillem García Subies	f5e2ed0fd8	Update tokenization_openai.py	2019-08-20 14:19:25 +02:00
Guillem García Subies	562b998366	Update tokenization_openai.py	2019-08-20 14:10:19 +02:00
Guillem García Subies	bb04446285	Update tokenization_openai.py	2019-08-20 14:07:40 +02:00
thomwolf	15d8b1266c	update tokenizer - update squad example for xlnet	2019-07-15 17:30:42 +02:00
thomwolf	b19786985d	unified tokenizer api and serialization + tests	2019-07-09 10:25:18 +02:00
thomwolf	36bca545ff	tokenization abstract class - tests for examples	2019-07-05 15:02:59 +02:00
thomwolf	0bab55d5d5	[BIG] name change	2019-07-05 11:55:36 +02:00