c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,751 Commits 171 Branches 49 Tags
73efa694e6cc9141ce732ea8215b25b3083298dc
Commit Graph
14 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Anthony MOI b7cf9f43d2 Update tokenizers to 0.7.0-rc5 (#3705) 2020-04-10 14:23:49 -04:00
Anthony MOI bfe870be65 Hotfix tokenizers version for sdist installs 2019-12-27 11:05:52 -05:00
Anthony MOI e6ec24fa88 Better added_tokens handling 2019-12-26 16:49:48 -05:00
Anthony MOI 599db139f9 Code style update 2019-12-26 15:13:30 -05:00
Anthony MOI 835b76a46f Handle unk_token 
As we discussed, this is handled here directly 
cc @thomwolf
 2019-12-26 14:42:55 -05:00
Anthony MOI 7ead04ce14 FastPreTrainedTokenizer => PreTrainedTokenizerFast 2019-12-26 14:39:39 -05:00
Anthony MOI 1f82a5d910 Update for changes in tokenizers API 2019-12-26 14:37:55 -05:00
Anthony MOI 734d29b03d tokenizers is now a real dependency 2019-12-24 13:32:41 -05:00
Anthony MOI 2818e50569 Add tests for fast tokenizers 2019-12-24 13:29:01 -05:00
Anthony MOI 31c56f2e0b Fix style 2019-12-24 12:43:27 -05:00
Anthony MOI 951ae99bea BertTokenizerFast 2019-12-24 12:24:24 -05:00
Anthony MOI 041eac2d6d GPT2TokenizerFast 2019-12-24 12:24:14 -05:00
Anthony MOI 3471ff0d35 FastPreTrainedTokenizer 2019-12-24 12:23:30 -05:00
Anthony MOI f2a3eb987e Fix small typos 2019-07-31 11:05:06 -04:00