c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,521 Commits 171 Branches 49 Tags
bd3feddf67f3005ca73c69abc07ffc5e64e8a0ba
Commit Graph
29 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Lysandre Debut 5164ea91a7 Skipping outputs (#3116) 
* Minimal example

* Proposal 2

* Proposal 2 for fast tokenizers

* Typings

* Docs

* Revert "Docs" for easier review

This reverts commit eaf0f97062e809887704a542144c537f769d5223.

* Remove unnecessary assignments

* Tests

* Fix faulty type

* Remove prints

* return_outputs -> model_input_names

* Revert "Revert "Docs" for easier review"

This reverts commit 6fdc69408102bf695797f2dfddbb6350c6b9e722.

* code quality
 2020-03-09 13:48:58 -04:00
Lysandre Debut 0ae91c80aa Change back pipeline signatures (#3105) 
* Change back pipeline signatures

* String types for non-imported objects
 2020-03-06 17:26:18 -05:00
Lysandre Debut fa2aa699da Merge pull request #3011 from patrickvonplaten/add_models_special_tokens_to_specific_configs 
Add models special tokens to its pretrained configs
 2020-03-05 17:26:48 -05:00
Lysandre Debut 146c521235 Merge branch 'master' into add_models_special_tokens_to_specific_configs 2020-03-05 17:24:42 -05:00
Lysandre Debut b623ddc000 Pass kwargs to configuration (#3147) 
* Pass kwargs to configuration

* Setter

* test
 2020-03-05 17:16:57 -05:00
Lysandre Debut 0001d05686 Correct missing keys + test (#3143) 2020-03-05 17:01:54 -05:00
Lysandre Debut d3eb7d23a4 Pipeline doc (#3055) 
* Pipeline doc initial commit

* pipeline abstraction

* Remove modelcard argument from pipeline

* Task-specific pipelines can be instantiated with no model or tokenizer

* All pipelines doc
 2020-03-02 14:07:10 -05:00
Lysandre Debut 8bcb37bfb8 NER support for Albert in run_ner.py and NerPipeline (#2983) 
* * Added support for Albert when fine-tuning for NER

* Added support for Albert in NER pipeline

* Added command-line options to examples/ner/run_ner.py to better control tokenization

* Added class AlbertForTokenClassification

* Changed output for NerPipeline to use .convert_ids_to_tokens(...) instead of .decode(...) to better reflect tokens

* Added ,

* Now passes style guide enforcement

* Changes from reviews.

* Code now passes style enforcement

* Added test for AlbertForTokenClassification

* Added test for AlbertForTokenClassification
 2020-02-27 10:22:55 -05:00
Lysandre Debut bb7c468520 Documentation (#2989) 
* All Tokenizers

BertTokenizer + few fixes
RobertaTokenizer
OpenAIGPTTokenizer + Fixes
GPT2Tokenizer + fixes
TransfoXLTokenizer
Correct rst for TransformerXL
XLMTokenizer + fixes
XLNet Tokenizer + Style
DistilBERT + Fix XLNet RST
CTRLTokenizer
CamemBERT Tokenizer
FlaubertTokenizer
XLMRobertaTokenizer
cleanup

* cleanup
 2020-02-25 18:43:36 -05:00
Lysandre Debutandjoshchagani 65e7c90a77 Adding usage examples for common tasks (#2850) 
* Usage: Sequence Classification & Question Answering

* Pipeline example

* Language modeling

* TensorFlow code for Sequence classification

* Custom TF/PT toggler in docs

* QA + LM for TensorFlow

* Finish Usage for both PyTorch and TensorFlow

* Addressing Julien's comments

* More assertive

* cleanup

* Favicon
- added favicon option in conf.py along with the favicon image
- udpated 🤗 logo. slightly smaller and should appear more consistent across editing programs (no more tongue on the outside of the mouth)

Co-authored-by: joshchagani <joshua@joshuachagani.com>
 2020-02-25 13:48:24 -05:00
Lysandre Debut b90745c590 Test correct tokenizers after default switch (#3003) 2020-02-24 18:45:53 -05:00
Lysandre Debut 3716c3d8af False by default (#3002) 2020-02-24 18:30:57 -05:00
Lysandre Debut 7984a70ee4 kwargs are passed to both model and configuration in AutoModels (#2998) 2020-02-24 14:19:39 -05:00
Lysandre Debut 21d8b6a33e Testing that batch_encode_plus is the same as encode_plus (#2973) 
* Testing that encode_plus and batch_encode_plus behave the same way

Spoiler alert: they don't

* Testing rest of arguments in batch_encode_plus

* Test tensor return in batch_encode_plus

* Addressing Sam's comments

* flake8

* Simplified with `num_added_tokens`
 2020-02-24 12:09:46 -05:00
Lysandre Debut 8194df8e0c Warning on add_special_tokens (#2966) 
Warning on `add_special_tokens` when passed to `encode`, `encode_plus` and `batch_encode_plus`
 2020-02-24 08:42:54 -05:00
Lysandre Debut 0e84559d64 Correct special_tokens_mask when add_special_tokens=False (#2965) 
Don't know of a use case where that would be useful, but this is more consistent
 2020-02-23 09:50:39 -05:00
Lysandre Debut 94ff2d6ee8 Remove double bias (#2958) 2020-02-21 17:10:18 -05:00
Lysandre Debut 33d3072e1c Arxiv README (#2747) 
* Arxiv README

* ArXiv-NLP readme
 2020-02-05 15:26:28 -05:00
Lysandre Debut 00c4e39581 Merge branch 'master' into squad-refactor 2019-12-09 10:41:15 -05:00
Lysandre Debut b632145273 Update master documentation link in README 2019-11-26 14:27:15 -05:00
Lysandre Debut 6e85bccafc Fixed typo 2019-10-22 18:07:01 -04:00
Lysandre Debut 8efc0ec91a Add Benchmarks to issue templates 2019-10-18 10:45:44 -04:00
Lysandre DebutandThomas Wolf @thomwolf c544194611 Remove special_tokens_mask from inputs in README 
Co-authored-by: Thomas Wolf @thomwolf
 2019-10-16 11:05:13 -04:00
Lysandre Debut 2431fea98a Merge pull request #1383 from keskarnitish/master 
Adding CTRL
 2019-10-09 11:31:05 -04:00
Lysandre Debut e84470ef81 Merge pull request #1384 from huggingface/encoding-qol 
Quality of life enhancements in encoding + patch MLM masking
 2019-10-09 11:18:24 -04:00
Lysandre Debut bb464289ce New model addition issue template 2019-10-04 16:41:26 -04:00
Lysandre Debut 81a1e12469 Merge pull request #1313 from enzoampil/master 
Add option to use a 'stop token'
 2019-10-03 22:43:57 +00:00
Lysandre Debut d3f24dfad7 Merge branch 'master' into master 2019-10-03 22:43:09 +00:00
Lysandre Debut 0f92f76ca3 CircleCI reference in README 2019-09-26 08:59:52 -04:00