Commit Graph
31 Commits
Author SHA1 Message Date
Lysandre Debut f049be7ad4 Export ALBERT main layer in TensorFlow (#3354) 2020-03-19 13:53:05 -04:00
Lysandre Debut d6afbd323d XLM-R Tokenizer now passes common tests + Integration tests (#3198)
* XLM-R now passes common tests + Integration tests

* Correct mask index

* Model input names

* Style

* Remove text preprocessing

* Unneccessary import
2020-03-18 09:52:49 -04:00
Lysandre Debut 5164ea91a7 Skipping outputs (#3116)
* Minimal example

* Proposal 2

* Proposal 2 for fast tokenizers

* Typings

* Docs

* Revert "Docs" for easier review

This reverts commit eaf0f97062e809887704a542144c537f769d5223.

* Remove unnecessary assignments

* Tests

* Fix faulty type

* Remove prints

* return_outputs -> model_input_names

* Revert "Revert "Docs" for easier review"

This reverts commit 6fdc69408102bf695797f2dfddbb6350c6b9e722.

* code quality
2020-03-09 13:48:58 -04:00
Lysandre Debut 0ae91c80aa Change back pipeline signatures (#3105)
* Change back pipeline signatures

* String types for non-imported objects
2020-03-06 17:26:18 -05:00
Lysandre Debut fa2aa699da Merge pull request #3011 from patrickvonplaten/add_models_special_tokens_to_specific_configs
Add models special tokens to its pretrained configs
2020-03-05 17:26:48 -05:00
Lysandre Debut 146c521235 Merge branch 'master' into add_models_special_tokens_to_specific_configs 2020-03-05 17:24:42 -05:00
Lysandre Debut b623ddc000 Pass kwargs to configuration (#3147)
* Pass kwargs to configuration

* Setter

* test
2020-03-05 17:16:57 -05:00
Lysandre Debut 0001d05686 Correct missing keys + test (#3143) 2020-03-05 17:01:54 -05:00
Lysandre Debut d3eb7d23a4 Pipeline doc (#3055)
* Pipeline doc initial commit

* pipeline abstraction

* Remove modelcard argument from pipeline

* Task-specific pipelines can be instantiated with no model or tokenizer

* All pipelines doc
2020-03-02 14:07:10 -05:00
Lysandre Debut 8bcb37bfb8 NER support for Albert in run_ner.py and NerPipeline (#2983)
* * Added support for Albert when fine-tuning for NER

* Added support for Albert in NER pipeline

* Added command-line options to examples/ner/run_ner.py to better control tokenization

* Added class AlbertForTokenClassification

* Changed output for NerPipeline to use .convert_ids_to_tokens(...) instead of .decode(...) to better reflect tokens

* Added ,

* Now passes style guide enforcement

* Changes from reviews.

* Code now passes style enforcement

* Added test for AlbertForTokenClassification

* Added test for AlbertForTokenClassification
2020-02-27 10:22:55 -05:00
Lysandre Debut bb7c468520 Documentation (#2989)
* All Tokenizers

BertTokenizer + few fixes
RobertaTokenizer
OpenAIGPTTokenizer + Fixes
GPT2Tokenizer + fixes
TransfoXLTokenizer
Correct rst for TransformerXL
XLMTokenizer + fixes
XLNet Tokenizer + Style
DistilBERT + Fix XLNet RST
CTRLTokenizer
CamemBERT Tokenizer
FlaubertTokenizer
XLMRobertaTokenizer
cleanup

* cleanup
2020-02-25 18:43:36 -05:00
Lysandre Debutandjoshchagani 65e7c90a77 Adding usage examples for common tasks (#2850)
* Usage: Sequence Classification & Question Answering

* Pipeline example

* Language modeling

* TensorFlow code for Sequence classification

* Custom TF/PT toggler in docs

* QA + LM for TensorFlow

* Finish Usage for both PyTorch and TensorFlow

* Addressing Julien's comments

* More assertive

* cleanup

* Favicon
- added favicon option in conf.py along with the favicon image
- udpated 🤗 logo. slightly smaller and should appear more consistent across editing programs (no more tongue on the outside of the mouth)

Co-authored-by: joshchagani <joshua@joshuachagani.com>
2020-02-25 13:48:24 -05:00
Lysandre Debut b90745c590 Test correct tokenizers after default switch (#3003) 2020-02-24 18:45:53 -05:00
Lysandre Debut 3716c3d8af False by default (#3002) 2020-02-24 18:30:57 -05:00
Lysandre Debut 7984a70ee4 kwargs are passed to both model and configuration in AutoModels (#2998) 2020-02-24 14:19:39 -05:00
Lysandre Debut 21d8b6a33e Testing that batch_encode_plus is the same as encode_plus (#2973)
* Testing that encode_plus and batch_encode_plus behave the same way

Spoiler alert: they don't

* Testing rest of arguments in batch_encode_plus

* Test tensor return in batch_encode_plus

* Addressing Sam's comments

* flake8

* Simplified with `num_added_tokens`
2020-02-24 12:09:46 -05:00
Lysandre Debut 8194df8e0c Warning on add_special_tokens (#2966)
Warning on `add_special_tokens` when passed to `encode`, `encode_plus` and `batch_encode_plus`
2020-02-24 08:42:54 -05:00
Lysandre Debut 0e84559d64 Correct special_tokens_mask when add_special_tokens=False (#2965)
Don't know of a use case where that would be useful, but this is more consistent
2020-02-23 09:50:39 -05:00
Lysandre Debut 94ff2d6ee8 Remove double bias (#2958) 2020-02-21 17:10:18 -05:00
Lysandre Debut 33d3072e1c Arxiv README (#2747)
* Arxiv README

* ArXiv-NLP readme
2020-02-05 15:26:28 -05:00
Lysandre Debut 00c4e39581 Merge branch 'master' into squad-refactor 2019-12-09 10:41:15 -05:00
Lysandre Debut b632145273 Update master documentation link in README 2019-11-26 14:27:15 -05:00
Lysandre Debut 6e85bccafc Fixed typo 2019-10-22 18:07:01 -04:00
Lysandre Debut 8efc0ec91a Add Benchmarks to issue templates 2019-10-18 10:45:44 -04:00
Lysandre DebutandThomas Wolf @thomwolf c544194611 Remove special_tokens_mask from inputs in README
Co-authored-by: Thomas Wolf @thomwolf
2019-10-16 11:05:13 -04:00
Lysandre Debut 2431fea98a Merge pull request #1383 from keskarnitish/master
Adding CTRL
2019-10-09 11:31:05 -04:00
Lysandre Debut e84470ef81 Merge pull request #1384 from huggingface/encoding-qol
Quality of life enhancements in encoding + patch MLM masking
2019-10-09 11:18:24 -04:00
Lysandre Debut bb464289ce New model addition issue template 2019-10-04 16:41:26 -04:00
Lysandre Debut 81a1e12469 Merge pull request #1313 from enzoampil/master
Add option to use a 'stop token'
2019-10-03 22:43:57 +00:00
Lysandre Debut d3f24dfad7 Merge branch 'master' into master 2019-10-03 22:43:09 +00:00
Lysandre Debut 0f92f76ca3 CircleCI reference in README 2019-09-26 08:59:52 -04:00