c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,145 Commits 171 Branches 49 Tags
6636826f0494a0ce81e33d5240e41d921f55cdcf
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Julien ChaumondandHenrykBorzymowski 6636826f04 [model_cards] Multilingual + Dutch SQuAD2.0 
Co-Authored-By: HenrykBorzymowski <henrykborzymowski@users.noreply.github.com>
 2020-01-31 18:39:09 -05:00
Julien ChaumondLoreto ParisiSimone Francia
 		98dadc98e1 [model_cards] UmBERTo 
Co-Authored-By: Loreto Parisi <loretoparisi@gmail.com>
Co-Authored-By: Simone Francia <francia.simone1@gmail.com>
 2020-01-31 18:39:09 -05:00
Julien Chaumond d6fc34b459 [model_cards] add mine 2020-01-31 18:39:09 -05:00
Julien Chaumond 5a6b138b00 [Umberto] model shortcuts (#2661) 
* [Umberto] model shortcuts

cc @loretoparisi @simonefrancia

see #2485

* Ensure that tokenizers will be correctly configured
 2020-01-30 21:05:53 -05:00
Julien Chaumond 7fe294bf07 Hotfix: same handling of non-existent files as for config 2020-01-30 20:05:04 -05:00
Julien Chaumond b85c59f997 config.architectures 2020-01-30 19:26:59 -05:00
Julien Chaumond f9bc3f5771 style tweak 2020-01-30 19:26:59 -05:00
Julien Chaumond 0b13fb822a No need for a model_type here 
cc @lysandrejik
 2020-01-30 19:26:59 -05:00
Julien Chaumond 9fa836a73f fill_mask helper (#2576) 
* fill_mask helper

* [poc] FillMaskPipeline

* Revert "[poc] FillMaskPipeline"

This reverts commit 67eeea55b0f97b46c2b828de0f4ee97d87338335.

* Revert "fill_mask helper"

This reverts commit cacc17b884e14bb6b07989110ffe884ad9e36eaa.

* README: clarify that Pipelines can also do text-classification

cf. question at the AI&ML meetup last week, @mfuntowicz

* Fix test: test feature-extraction pipeline

* Test tweaks

* Slight refactor of existing pipeline (in preparation of new FillMaskPipeline)

* Extraneous doc

* More robust way of doing this

@mfuntowicz as we don't rely on the model name anymore (see AutoConfig)

* Also add RobertaConfig as a quickfix for wrong token_type_ids

* cs

* [BIG] FillMaskPipeline
 2020-01-30 18:15:42 -05:00
Julien Chaumond 6b4c3ee234 [run_lm_finetuning] GPT2 tokenizer doesn't have a pad_token 
ping @lysandrejik
 2020-01-27 20:14:02 -05:00
Julien Chaumond 79815bf666 [serving] Fix typo 2020-01-27 19:58:25 -05:00
Julien Chaumond 5004d5af42 [serving] Update dependencies 2020-01-27 19:58:00 -05:00
Julien Chaumond ea56d305be make style 2020-01-27 12:13:32 -05:00
Julien Chaumond 11b13e94a3 Add type to help my IDE out 2020-01-24 14:00:57 -05:00
Julien Chaumond 90b7df444f Upload CLI: on win32, use slashes, not os.sep 2020-01-22 22:41:21 -05:00
Julien Chaumond 119dc50e2a Doc tweak on model sharing 2020-01-22 22:40:38 -05:00
Julien Chaumond 34a3c25a30 Fix for XLMRobertaConfig inherits from RobertaConfig 
hat/tip @stefan-it
 2020-01-22 17:50:24 -05:00
Julien Chaumond 1a8e87be4e Line-by-line text dataset (including padding) 2020-01-21 16:57:38 -05:00
Julien Chaumond b94cf7faac change order 2020-01-21 16:57:38 -05:00
Julien Chaumond 2eaa8b6e56 Easier to not support this, as it could be confusing 
cc @lysandrejik
 2020-01-21 16:57:38 -05:00
Julien Chaumond 801aaa5508 make style 2020-01-21 16:57:38 -05:00
Julien Chaumond 56d4ba8ddb [run_lm_finetuning] Train from scratch 2020-01-21 16:57:38 -05:00
Julien Chaumond 23a2cea8cb Tokenizer.from_pretrained: fetch all possible files remotely 2020-01-16 16:47:19 -05:00
Julien Chaumond 99f9243de5 same here, try to not serialize too much if unneeded 2020-01-16 16:47:19 -05:00
Julien Chaumond 9d8fd2d40e tokenizer.save_pretrained: only save file if non-empty 2020-01-16 16:47:19 -05:00
Julien Chaumond d9fa1bad72 Fix failing torchscript test for xlnet 
model.parameters() order is apparently not stable (only for xlnet, for some reason)
 2020-01-15 20:22:21 -05:00
Julien Chaumond a98b2ca8c0 Style + fixup BertJapaneseTokenizer 2020-01-15 19:05:51 -05:00
Julien Chaumond 83a41d39b3 💄 super 2020-01-15 18:33:50 -05:00
Julien Chaumond cd51893d37 Merge branch 'Rexhaif-patch-1' 2020-01-15 18:25:15 -05:00
Julien Chaumond 248aeaa842 Merge branch 'patch-1' of https://github.com/Rexhaif/transformers into Rexhaif-patch-1 2020-01-15 18:22:01 -05:00
Julien Chaumond eb59e9f705 Graduate sst-2 to a canonical one 2020-01-15 16:28:50 +00:00
Julien Chaumond e184ad13cf Close #2392 2020-01-15 15:43:44 +00:00
Julien Chaumond 715fa638a7 Merge branch 'master' into from_scratch_training 2020-01-14 18:58:21 +00:00
Julien Chaumond 764f836d52 Update test_tokenization_auto.py 2020-01-13 22:50:34 -05:00
Julien Chaumond d5831acb07 Update test_tokenization_auto.py 2020-01-13 22:47:33 -05:00
Julien Chaumond ed6cd597cc Update test_tokenization_auto.py 2020-01-13 22:46:35 -05:00
Julien Chaumond 5cb463a714 Update test_tokenization_auto.py 2020-01-13 22:38:29 -05:00
Julien Chaumond afc24ea5d4 In a parallel setup this could fail 2020-01-13 23:44:08 +00:00
Julien Chaumond 894812c652 Fixup mapping 2020-01-13 23:34:19 +00:00
Julien Chaumond b20f11d4ca 🔫 Python35 2020-01-13 23:20:44 +00:00
Julien Chaumond 0304628590 Map configs to models and tokenizers 2020-01-13 23:11:44 +00:00
Julien Chaumond 1fc855e456 [tests] Safety checks on CONFIG_MAPPING 2020-01-13 21:52:55 +00:00
Julien Chaumond 3c86b6f3c5 Py35 doesn't like inline variable types 2020-01-13 20:44:33 +00:00
Julien Chaumond b803b067bf Config to Model mapping 2020-01-13 20:05:20 +00:00
Julien Chaumond cf8a70bf68 More AutoConfig tests 2020-01-11 03:43:57 +00:00
Julien Chaumond 6bb3edc300 Serialize model_type if exists 2020-01-11 03:18:56 +00:00
Julien Chaumond c6f682c1eb flake 2020-01-11 03:18:31 +00:00
Julien Chaumond 4d1c98c012 AutoConfig + other Auto classes honor model_type 2020-01-11 02:46:17 +00:00
Julien Chaumond 2f32dfd33b Convention: name mixins mixins 2020-01-11 01:24:29 +00:00
Julien Chaumond 055e80cfad rm old ConfigTester 2020-01-10 21:36:18 +00:00
Julien Chaumond fd8423321f keep list sorted 2020-01-10 20:36:46 +00:00
Julien Chaumond 0cd81fb99f [isort] declare more third-parties in case no tf install 2020-01-10 20:35:45 +00:00
Julien Chaumond 84c0aa1868 num_parameters helper 2020-01-10 17:40:02 +00:00
Julien Chaumond f599623a99 PreTrainedTokenizerFast: hotfix _convert_encoding 
cc @n1t0
 2020-01-08 15:46:37 -05:00
Julien Chaumond c301faa92b Distributed or parallel setup 2020-01-06 18:41:08 -05:00
Julien Chaumond 78528742f1 Fix syntax + link to community page 2020-01-05 12:43:39 -05:00
Julien Chaumond 629b22adcf [run_lm_finetuning] mask_tokens: document types 2020-01-01 12:55:10 -05:00
Julien Chaumond 594ca6dead [debug] Debug Heisenbug, the old school way. 2019-12-29 10:07:21 -05:00
Julien Chaumond 0df4e62da0 [http] Tweak http user-agent (#2353) 2019-12-29 10:06:50 -05:00
Julien Chaumond 0d467fd6de Typo 2019-12-27 23:06:48 -05:00
Julien Chaumond d8293e84f3 [cli] upload: max number of files at the same time 2019-12-27 23:02:53 -05:00
Julien Chaumond 4d6c93e923 Kill __main__ 2019-12-27 22:55:22 -05:00
Julien Chaumond 9b2badf3c9 [cli] Update doc 2019-12-27 22:54:29 -05:00
Julien Chaumond f78ebc22ad [cli] Add ability to delete remote object 2019-12-27 22:53:49 -05:00
Julien Chaumond ac1b449cc9 [doc] move distilroberta to more appropriate place 
cc @lysandrejik
 2019-12-21 00:09:01 -05:00
Julien Chaumond 3e52915fa7 [RoBERTa] Embeddings: fix dimensionality bug 2019-12-20 19:01:27 -05:00
Julien Chaumond 15d897ff4a [http] customizable requests user-agent 2019-12-19 18:29:22 -05:00
Julien Chaumond f25e9b6f77 [hf_bucket_url] support for cloudfront urls 2019-12-19 18:28:17 -05:00
Julien Chaumond a5a06a851e [doc] Param name consistency 2019-12-19 16:24:20 -05:00
Julien Chaumond 9a399ead25 Revert incorrect #1778 2019-12-19 15:45:48 -05:00
Julien Chaumond 94c99db34c [FinBERT] fix incorrect url 2019-12-17 20:35:25 -05:00
Julien Chaumond 7ffa817390 [s3] mv files and update links 2019-12-17 20:35:25 -05:00
Julien Chaumond a0d386455b Fix outdated tokenizer doc 2019-12-17 20:07:39 -05:00
Julien Chaumond ea636440d1 [roberta.conversion] Do not hardcode vocab size 
and support for fairseq 0.9+
 2019-12-17 18:12:22 -05:00
Julien Chaumond 55397dfb9b CsvPipelineDataFormat: Fix for single-column 2019-12-17 13:10:51 -05:00
Julien Chaumond 3f5ccb183e [doc] Clarify uploads 
cf https://github.com/huggingface/transformers/commit/855ff0e91d8b3bd75a3b1c1316e2efd814373764#commitcomment-36452545
 2019-12-16 18:20:29 -05:00
Julien Chaumond 855ff0e91d [doc] Model upload and sharing 
ping @lysandrejik @thomwolf

Is this clear enough? Anything we should add?
 2019-12-16 12:42:22 -05:00
Julien Chaumond b6d4284b26 [cli] Uploads: fix + test edge case 2019-12-13 22:44:57 -05:00
Julien Chaumond 866d73ca26 [cli] Upload is now compatible with folders 2019-12-13 16:39:08 -05:00
Julien Chaumond 5d67aa21ae [doc] Replicate doc from #2144 2019-12-12 12:39:41 -05:00
Julien Chaumond 1748fdf657 [doc] Fix rst table 2019-12-11 18:32:27 -05:00
Julien Chaumond 36fc52a3b4 Update links to weights 2019-12-11 18:32:27 -05:00
Julien Chaumond 371c5ddfad Py2 tests for Lysandre 2019-12-11 18:32:27 -05:00
Julien Chaumond 5505cf7014 Run tests on Py2 too, for Lysandre 2019-12-11 18:32:27 -05:00
Julien Chaumond 9cb97c0c0f Actually run the tests 2019-12-11 18:32:27 -05:00
Julien Chaumond 95854c4a2f Actually run the tests 2019-12-11 18:32:27 -05:00
Julien Chaumond d2100428d3 Update to new test infra and only run conditionally 2019-12-11 18:32:27 -05:00
Julien Chaumond 4f15e5a267 Add tests. 
Maybe not the best possible place for the tests, lmk.
 2019-12-11 17:41:51 -05:00
Julien Chaumond 18e1f751f1 TF support 2019-12-11 17:07:46 -05:00
Julien Chaumond 31e5b5ff22 Fix tests + first example of doc 2019-12-11 15:22:02 -05:00
Julien Chaumond c999a3e505 Allow from_pretrained to take a remote identifier 2019-12-11 12:29:58 -05:00
Julien Chaumond 2e2f9fed55 rm duplicate imports 2019-12-11 11:11:56 -05:00
Julien Chaumond 1d18930462 Harmonize no_cuda flag with other scripts 2019-12-09 20:37:55 -05:00
Julien Chaumond e4679cddce [cli] Uploads: add progress bar (#2078) 
* [cli] Uploads: add progress bar

see https://github.com/huggingface/transformers/pull/2044#discussion_r354057827 for context

* rename + documentation

* Add auto-referential comment
 2019-12-06 11:56:23 -05:00
Julien Chaumond 3ba417e1a8 [cli] ls: Tabular formatting 2019-12-04 18:40:52 -05:00
Julien Chaumond 96fa9a8a70 Python 2 + Post mime-type to S3 2019-12-04 17:22:50 -05:00
Julien Chaumond e4fbf3e2cc CLI for authenticated file sharing 2019-12-04 00:52:23 -05:00
Julien Chaumond 7edb51f3a5 [pplm] split classif head into its own file 2019-12-03 22:07:25 +00:00
Julien ChaumondandPiero Molino f434bfc623 [pplm] Update S3 links 
Co-Authored-By: Piero Molino <w4nderlust@gmail.com>
 2019-12-03 10:53:02 -05:00
Julien Chaumond 3b48806f75 [pplm] README: add setup + tweaks 2019-12-03 10:14:02 -05:00