5 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Pascal Voitot	f2ac50cb55	better for python2.x	2019-12-13 16:41:44 -05:00
Pascal Voitot	4cbdc7d910	missed space	2019-12-13 16:41:44 -05:00
Pascal Voitot	dd2add9f6e	more tests	2019-12-13 16:41:44 -05:00
Pascal Voitot	df160af736	🐛 #2096 in tokenizer.decode, space is not joined between all subtexts instead of before added tokens	2019-12-13 16:41:44 -05:00
Pascal Voitot	5b7b78e088	🐛 #2096 in tokenizer.decode, adds a space after special tokens to return right formatted string	2019-12-13 16:41:44 -05:00