c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,949 Commits 171 Branches 49 Tags
af1ee9e648e644aae45c5250a46d4c25a4e0d04d
Commit Graph
13 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Anthony MOI bfe870be65 Hotfix tokenizers version for sdist installs 2019-12-27 11:05:52 -05:00
Anthony MOI e6ec24fa88 Better added_tokens handling 2019-12-26 16:49:48 -05:00
Anthony MOI 599db139f9 Code style update 2019-12-26 15:13:30 -05:00
Anthony MOI 835b76a46f Handle unk_token 
As we discussed, this is handled here directly 
cc @thomwolf
 2019-12-26 14:42:55 -05:00
Anthony MOI 7ead04ce14 FastPreTrainedTokenizer => PreTrainedTokenizerFast 2019-12-26 14:39:39 -05:00
Anthony MOI 1f82a5d910 Update for changes in tokenizers API 2019-12-26 14:37:55 -05:00
Anthony MOI 734d29b03d tokenizers is now a real dependency 2019-12-24 13:32:41 -05:00
Anthony MOI 2818e50569 Add tests for fast tokenizers 2019-12-24 13:29:01 -05:00
Anthony MOI 31c56f2e0b Fix style 2019-12-24 12:43:27 -05:00
Anthony MOI 951ae99bea BertTokenizerFast 2019-12-24 12:24:24 -05:00
Anthony MOI 041eac2d6d GPT2TokenizerFast 2019-12-24 12:24:14 -05:00
Anthony MOI 3471ff0d35 FastPreTrainedTokenizer 2019-12-24 12:23:30 -05:00
Anthony MOI f2a3eb987e Fix small typos 2019-07-31 11:05:06 -04:00