 Shijie Wu
|
a15562e170
|
Fix reference of import when called for the second time
|
2019-09-03 18:27:29 -07:00 |
|
|
Shijie Wu
|
ca4baf8ca1
|
Match order of casing in OSS XLM; Improve document; Clean up dependency
|
2019-08-27 20:03:18 -04:00 |
|
|
Shijie Wu
|
f1b018740c
|
Add use_lang_emb to config
|
2019-08-23 20:33:01 -04:00 |
|
|
Shijie Wu
|
e85123d398
|
Add custom tokenizer for zh and ja
|
2019-08-23 20:27:52 -04:00 |
|
|
Shijie Wu
|
436ce07218
|
Tokenization behave the same as original XLM proprocessing for most languages except zh, ja and th; Change API to allow specifying language in tokenize
|
2019-08-23 14:40:17 -04:00 |
|