3 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Yiqing-Zhou	b1019d2a8e	token[-1] -> token.rstrip('\n')	2019-07-23 20:41:26 +08:00
Yiqing-Zhou	bef0c629ca	fix ... Remove '\n' before adding token into vocab	2019-07-22 22:30:49 +08:00
Yiqing-Zhou	897d0841be	read().splitlines() -> readlines() ... splitlines() does not work as what we expect here for bert-base-chinese because there is a '\u2028' (unicode line seperator) token in vocab file. Value of '\u2028'.splitlines() is ['', '']. Perhaps we should use readlines() instead.	2019-07-22 20:49:09 +08:00