c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,762 Commits 171 Branches 49 Tags
15b9868f8b33890f675581a6617c76e200643bbc
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Thomas WolfandStefan Schweter 827d6d6ef0 Cleanup fast tokenizers integration (#3706) 
* First pass on utility classes and python tokenizers

* finishing cleanup pass

* style and quality

* Fix tests

* Updating following @mfuntowicz comment

* style and quality

* Fix Roberta

* fix batch_size/seq_length inBatchEncoding

* add alignement methods + tests

* Fix OpenAI and Transfo-XL tokenizers

* adding trim_offsets=True default for GPT2 et RoBERTa

* style and quality

* fix tests

* add_prefix_space in roberta

* bump up tokenizers to rc7

* style

* unfortunately tensorfow does like these - removing shape/seq_len for now

* Update src/transformers/tokenization_utils.py

Co-Authored-By: Stefan Schweter <stefan@schweter.it>

* Adding doc and docstrings

* making flake8 happy

Co-authored-by: Stefan Schweter <stefan@schweter.it>
 2020-04-18 13:43:57 +02:00
Thomas Wolf 2187c49f5c CPU/GPU memory benchmarking utilities - Remove support for python 3.5 (now only 3.6+) (#3186) 
* memory benchmark rss

* have both forward pass and line-by-line mem tracing

* cleaned up tracing

* refactored and cleaning up API

* no f-strings yet...

* add GPU mem logging

* fix GPU memory monitoring

* style and quality

* clean up and doc

* update with comments

* Switching to python 3.6+

* fix quality
 2020-03-17 10:17:11 -04:00
Thomas Wolf 3814e167d9 Merge pull request #3225 from patrickvonplaten/finalize_merge_bart_generate_into_default_generate 
Complete merge Seq-2-Seq generation into default generation
 2020-03-14 15:08:59 +01:00
Thomas Wolf db29ffc978 Merge pull request #3140 from patrickvonplaten/merge_bart_generate_into_default_generate 
Merge bart generate into default generate
 2020-03-11 13:21:53 +01:00
Thomas Wolf 0416d437fb Merge pull request #3148 from patrickvonplaten/refactoring_beam_search_for_tf_2 
refactored beam search according to torch implementation
 2020-03-06 22:01:46 +01:00
Thomas Wolf 6ffe03a0a1 Merge pull request #3137 from tomhosking/bart-refactor 
Refactor BartModel so that input checks are handled within enc/dec
 2020-03-06 13:06:34 +01:00
Thomas Wolf 3e5da38dae Merge pull request #3132 from huggingface/hf_api_model_list 
[hf_api] Get the public list of all the models on huggingface
 2020-03-06 13:05:52 +01:00
Thomas Wolf 9499a3778e Merge pull request #3103 from gthb/keras-serialization 
Support keras JSON/HDF5 serialization of main layers
 2020-03-06 12:59:13 +01:00
Thomas Wolf 1741d740f2 Merge pull request #3145 from sshleifer/bartfp16 
[Bart] FP16 Support
 2020-03-05 22:14:35 +01:00
Thomas Wolf bbabbc1613 Merge pull request #3135 from patrickvonplaten/refactor_beam_search_generate 
Refactoring and bug fixing beam search generate
 2020-03-05 22:12:56 +01:00
Thomas Wolf bdd3d0c76d Merge pull request #3118 from patrickvonplaten/add_beam_search_to_generation_tf_2_0 
Add beam search to generation tf 2 0
 2020-03-04 23:28:00 +01:00
Thomas Wolf 3b7f95a506 Merge pull request #3115 from gthb/fix-bogus-param-to-layer-init 
fix: passing config as Layer trainable param
 2020-03-04 21:59:09 +01:00
Thomas Wolf 5b2d4f2657 Merge pull request #2881 from patrickvonplaten/add_vim_swp_to_gitignore 
update .gitignore to ignore .swp files created when using vim
 2020-02-17 14:36:49 +01:00
Thomas Wolf 92e974196f Merge pull request #2765 from huggingface/extract-cached-archives 
Add option to `cached_path` to automatically extract archives
 2020-02-10 14:05:16 +01:00
Thomas Wolf b8f43cb273 Merge pull request #2239 from ns-moosavi/HANS-evaluation-example 
HANS evaluation
 2020-01-16 13:28:25 +01:00
Thomas Wolf dc17f2a111 Merge pull request #2538 from huggingface/py3_super 
💄 super
 2020-01-16 13:17:15 +01:00
Thomas Wolf 880854846b Merge pull request #2540 from huggingface/torch14_fix 
[PyTorch 1.4] Fix failing torchscript test for xlnet
 2020-01-16 13:16:59 +01:00
Thomas Wolf 2b566c182e Merge pull request #2384 from dimagalat/master 
Releasing file lock
 2020-01-14 13:19:01 +01:00
Thomas Wolf 896a0eb1fd Merge pull request #2459 from Perseus14/patch-4 
Update pipelines.py
 2020-01-13 16:02:54 +01:00
Thomas Wolf b1e1a9f9b2 Merge pull request #2495 from mschrimpf/patch-1 
T5: move rp_bucket to relative_attention_bias' device
 2020-01-10 22:18:54 +01:00
Thomas Wolf f75bf05ce6 Merge pull request #2352 from huggingface/cli_tweaks 
Cli tweaks
 2019-12-28 15:40:00 +01:00
Thomas Wolf 74ea432847 Merge pull request #2286 from adelevie/patch-2 
Typo in tokenization_utils.py
 2019-12-27 10:50:47 +01:00
Thomas Wolf 492bea9aa0 Merge pull request #2292 from patrickvonplaten/add_cached_past_for_language_generation 
Add cached past for language generation
 2019-12-27 10:33:27 +01:00
Thomas Wolf e213900fa2 Merge pull request #2290 from patrickvonplaten/fix_typo_in_doc_for_language_generation 
duplicated line for repeating_words_penalty_for_language_generation
 2019-12-27 10:29:06 +01:00
Thomas Wolf 9f5f646442 Merge pull request #2211 from huggingface/fast-tokenizers 
Fast tokenizers
 2019-12-27 10:24:29 +01:00
Thomas Wolf 8c67b529f6 Merge pull request #2324 from kashif/patch-1 
Typo in serving.py
 2019-12-26 12:38:06 +01:00
Thomas Wolf aeef4823ab Merge pull request #2303 from patrickvonplaten/fix_error_with_repetition_penalty 
fix repetition penalty error in modeling_utils.py
 2019-12-25 22:39:20 +01:00
Thomas Wolf 0412f3d929 Merge pull request #2291 from aaugustin/fix-flake8-F841 
Fix F841 flake8 warning
 2019-12-25 22:37:42 +01:00
Thomas Wolf 8742c95461 Merge pull request #2289 from patrickvonplaten/fix_effective_batch_size_lang_gen_xlm 
fix bug in prepare inputs for language generation for xlm for effective batch_size > 1
 2019-12-25 22:30:46 +01:00
Thomas Wolf 1240be3ed9 Merge pull request #2312 from vitaliyradchenko/fix_special_and_add_tokens_loading 
Correct tokenization for special and added tokens
 2019-12-25 20:52:30 +01:00
Thomas Wolf cea04a2443 Merge pull request #2310 from ShnitzelKiller/scatter-unfix 
revert erroneous fix #2276
 2019-12-25 12:43:22 +01:00
Thomas Wolf 81db12c3ba Merge pull request #2271 from aaugustin/improve-setup-and-requirements 
Improve setup and requirements
 2019-12-24 11:21:20 +01:00
Thomas Wolf ba2378ced5 Merge pull request #2264 from upura/fix-doclink 
Fix doc link in README
 2019-12-23 12:31:00 +01:00
Thomas Wolf e4e2a666c9 Merge pull request #2276 from ShnitzelKiller/scatterfix 
fix error due to wrong argument name to Tensor.scatter()
 2019-12-23 12:19:48 +01:00
Thomas Wolf 5daca95ddd Merge pull request #2268 from aaugustin/improve-repository-structure 
Improve repository structure
 2019-12-22 16:41:53 +01:00
Thomas Wolf 54abc67aec Merge pull request #2255 from aaugustin/implement-best-practices 
Implement some Python best practices
 2019-12-22 16:31:11 +01:00
Thomas Wolf 645713e2cb Merge pull request #2254 from huggingface/fix-tfroberta 
adding positional embeds masking to TFRoBERTa
 2019-12-21 15:33:22 +01:00
Thomas Wolf 73f6e9817c Merge pull request #2115 from suvrat96/add_mmbt_model 
[WIP] Add MMBT Model to Transformers Repo
 2019-12-21 15:26:08 +01:00
Thomas Wolf 5b7fb6a4a1 Merge pull request #2134 from bkkaggle/saving-and-resuming 
closes #1960 Add saving and resuming functionality for remaining examples
 2019-12-21 15:03:53 +01:00
Thomas Wolf 6f68d559ab Merge pull request #2130 from huggingface/ignored-index-coherence 
[BREAKING CHANGE] Setting all ignored index to the PyTorch standard
 2019-12-21 14:55:40 +01:00
Thomas Wolf 518ba748e0 Merge branch 'master' into saving-and-resuming 2019-12-21 14:41:39 +01:00
Thomas Wolf 18601c3b6e Merge pull request #2173 from erenup/master 
run_squad with roberta
 2019-12-21 14:33:16 +01:00
Thomas Wolf 6e7102cfb3 Merge pull request #2203 from gthb/patch-1 
fix: wrong architecture count in README
 2019-12-21 14:31:44 +01:00
Thomas Wolf deceb00161 Merge pull request #2177 from mandubian/issue-2106 
:zip: #2106 tokenizer.tokenize speed improvement (3-8x) by caching added_tokens in a Set
 2019-12-21 14:31:20 +01:00
Thomas Wolf eeb70cdd77 Merge branch 'master' into saving-and-resuming 2019-12-21 14:29:59 +01:00
Thomas Wolf ed9b84816e Merge pull request #1840 from huggingface/generation_sampler 
[WIP] Sampling sequence generator for transformers
 2019-12-21 14:27:35 +01:00
Thomas Wolf 7e17f09fb5 Merge pull request #1803 from importpandas/fix-xlnet-squad2.0 
fix run_squad.py during fine-tuning xlnet on squad2.0
 2019-12-21 13:38:48 +01:00
Thomas Wolf 562f864038 Merge branch 'master' into fix-xlnet-squad2.0 2019-12-21 12:48:10 +01:00
Thomas Wolf 8618bf15d6 Merge pull request #1736 from huggingface/fix-tf-xlnet 
Fix TFXLNet
 2019-12-21 12:42:05 +01:00
Thomas Wolf 2fa8737c44 Merge pull request #1586 from enzoampil/include_special_tokens_in_bert_examples 
Add special tokens to documentation for bert examples to resolve issue: #1561
 2019-12-21 12:36:11 +01:00
Thomas Wolf f15f087143 Merge pull request #1764 from DomHudson/bug-fix-1761 
Bug-fix: Roberta Embeddings Not Masked
 2019-12-21 12:13:27 +01:00
Thomas Wolf fae4d1c266 Merge pull request #2217 from aaugustin/test-parallelization 
Support running tests in parallel
 2019-12-21 11:54:23 +01:00
Thomas Wolf d0f8b9a978 Merge pull request #2244 from huggingface/fix-tok-pipe 
Fix Camembert and XLM-R `decode` method- Fix NER pipeline alignement
 2019-12-20 22:10:39 +01:00
Thomas Wolf a557836a70 Merge pull request #2191 from huggingface/fix_sp_np 
Numpy compatibility for sentence piece
 2019-12-20 22:08:08 +01:00
Thomas Wolf 8d5a47c79b Merge pull request #2243 from huggingface/fix-xlm-roberta 
fixing xlm-roberta tokenizer max_length and automodels
 2019-12-20 19:34:08 +01:00
Thomas Wolf 3a2c4e6f63 Merge pull request #1548 from huggingface/cli 
[2.2] - Command-line interface - Pipeline class
 2019-12-20 15:28:29 +01:00
Thomas Wolf 59941c5d1f Merge pull request #2189 from stefan-it/xlmr 
Add support for XLM-RoBERTa
 2019-12-20 13:26:38 +01:00
Thomas Wolf ff36e6d8d7 Merge pull request #2231 from huggingface/requests_user_agent 
[http] customizable requests user-agent
 2019-12-20 10:28:10 +01:00
Thomas Wolf f061606277 Merge pull request #2164 from huggingface/cleanup-configs 
[SMALL BREAKING CHANGE] Cleaning up configuration classes - Adding Model Cards
 2019-12-17 09:10:16 +01:00
Thomas Wolf a52d56c8d9 Merge branch 'master' into cleanup-configs 2019-12-14 09:43:07 +01:00
Thomas Wolf e92bcb7eb6 Merge pull request #1739 from huggingface/t5 
[WIP] Adding Google T5 model
 2019-12-14 09:40:43 +01:00
Thomas Wolf 110394b2ba Merge branch 'master' into t5 2019-12-13 16:03:32 +01:00
Thomas Wolf 90df44f0aa Merge pull request #2063 from guillaume-be/special_tokens_mask_value_not_used 
special_tokens_mask value was unused and calculated twice
 2019-12-12 08:21:46 +01:00
Thomas Wolf 707f9e9241 Merge pull request #2081 from pglock/patch-1 
handle string with only whitespaces as empty
 2019-12-12 08:20:43 +01:00
Thomas Wolf 137e20a846 Merge pull request #2075 from huggingface/check-link-validity 
Check link validity
 2019-12-12 08:09:12 +01:00
Thomas Wolf d5712f7cac Merge branch 'master' into check-link-validity 2019-12-12 08:00:51 +01:00
Thomas Wolf 9c58b236ef Merge pull request #2144 from huggingface/from-pretrained-from-url 
Allowing from_pretrained to load from url directly
 2019-12-12 07:43:40 +01:00
Thomas Wolf 386a93f0f8 Merge branch 'master' into from-pretrained-from-url 2019-12-12 07:31:05 +01:00
Thomas Wolf 2d103546ef Merge pull request #2148 from huggingface/fix_encode_plus 
Fix encode plus
 2019-12-12 07:24:47 +01:00
Thomas Wolf 51ae203290 Merge pull request #2129 from leopd/master 
Progress indicator improvements when downloading pre-trained models.
 2019-12-10 22:18:55 +01:00
Thomas Wolf e6cff60b4c Merge pull request #2069 from huggingface/cleaner-pt-tf-conversion 
clean up PT <=> TF conversion
 2019-12-10 15:34:08 +01:00
Thomas Wolf e57d00ee10 Merge pull request #1984 from huggingface/squad-refactor 
[WIP] Squad refactor
 2019-12-10 11:07:26 +01:00
Thomas Wolf ecabbf6d28 Merge pull request #2107 from huggingface/encoder-mask-shape 
create encoder attention mask from shape of hidden states
 2019-12-10 10:07:56 +01:00
Thomas Wolf 4cb9b60558 Merge pull request #2077 from patrickvonplaten/change_documentation_for_past_output_shape 
corrected documentation for past tensor shape for ctrl and gpt2 model
 2019-12-06 12:14:48 +01:00
Thomas Wolf 5482822a2b Merge pull request #2046 from jplu/tf2-ner-example 
Add NER TF2 example.
 2019-12-06 12:12:22 +01:00
Thomas Wolf fc1bb1f867 Merge pull request #2068 from huggingface/fix-2042 
Nicer error message when Bert's input is missing batch size
 2019-12-06 12:06:42 +01:00
Thomas Wolf bebaa14039 Merge pull request #2045 from aaugustin/remove-dead-code 
Remove dead code in tests.
 2019-12-05 14:41:56 +01:00
Thomas Wolf af077b15e2 Merge pull request #2065 from huggingface/fixing-camembert 
Fixing camembert tokenization
 2019-12-05 13:45:44 +01:00
Thomas Wolf 1f179f095f Merge pull request #2011 from AdityaSoni19031997/patch-1 
typo fix on the docs as per Pytorch v1.1+
 2019-12-05 12:39:04 +01:00
Thomas Wolf 1eaf44e713 Merge pull request #2007 from roskoN/xlnet_attention_fix 
fixed XLNet attention output for both attention streams whenever target_mapping is provided
 2019-12-05 12:32:39 +01:00
Thomas Wolf f9f395b21c Merge pull request #1735 from ondewo/tf-do-not-use-gpu-on-import 
Do not use GPU when importing transformers
 2019-12-05 11:56:48 +01:00
Thomas Wolf d425a4d60b Merge pull request #1870 from alexzubiaga/xlnet-for-token-classification 
XLNet for Token classification
 2019-12-05 09:54:09 +01:00
Thomas Wolf 1eb89ddf73 Merge pull request #2044 from huggingface/cli_upload 
CLI for authenticated file sharing
 2019-12-05 09:44:07 +01:00
Thomas Wolf cae641ff26 Merge pull request #1846 from tamuhey/patch/iss1845 
fix summary_type value of SequenceSummary
 2019-12-04 13:28:39 +01:00
Thomas Wolf f19a78a634 Merge pull request #1903 from valohai/master 
Valohai integration
 2019-12-03 16:13:01 +01:00
Thomas Wolf d100ad99c0 Merge pull request #2014 from aaugustin/mark-tf-auto-model-test-as-slow 
Mark tests in TFAutoModelTest as slow.
 2019-12-03 16:03:48 +01:00
Thomas Wolf 1ab8dc44b3 Merge pull request #1876 from huggingface/mean-fix 
Mean does not exist in TF2
 2019-11-29 09:26:33 +01:00
Thomas Wolf f0d22b6363 Merge pull request #1873 from stefan-it/distilbert-german 
German DistilBERT
 2019-11-29 09:25:47 +01:00
Thomas Wolf d49c43ff78 Merge pull request #1778 from eukaryote31/patch-2 
from_pretrained: convert DialoGPT format
 2019-11-28 16:08:37 +01:00
Thomas Wolf 91caf2462c Merge pull request #1770 from huggingface/initi-encoder-mask 
Only init encoder_attention_mask if stack is decoder
 2019-11-28 16:06:55 +01:00
Thomas Wolf 49a69d5b78 Merge pull request #1753 from digantamisra98/patch-1 
Added Mish Activation Function
 2019-11-28 15:24:08 +01:00
Thomas Wolf 96e7ee7238 Merge pull request #1740 from huggingface/fix-ctrl-past 
Fix CTRL past
 2019-11-27 23:28:30 +01:00
Thomas Wolf a36f981d1b Merge branch 'master' into fix-ctrl-past 2019-11-27 17:25:46 +01:00
Thomas Wolf 5afca00b47 Merge pull request #1724 from huggingface/fix_encode_plus 
Fix encode_plus
 2019-11-27 17:14:49 +01:00
Thomas Wolf 49108288ba Merge pull request #1624 from Huawei-MRC-OSI/resumable_http 
Add support for resumable downloads for HTTP protocol.
 2019-11-27 17:11:07 +01:00
Thomas Wolf 5340d1f21f Merge branch 'master' into resumable_http 2019-11-27 17:10:36 +01:00
Thomas Wolf 7fd1d42a01 Merge pull request #1592 from watkinsm/do_lower_case 
Consider do_lower_case in PreTrainedTokenizer
 2019-11-27 17:05:18 +01:00
Thomas Wolf 21637d4924 Merge branch 'master' into do_lower_case 2019-11-27 17:04:39 +01:00
Thomas Wolf 0cdfcca24b Merge pull request #1860 from stefan-it/camembert-for-token-classification 
[WIP] Add support for CamembertForTokenClassification
 2019-11-21 10:56:07 +01:00
Thomas Wolf 74ce8de7d8 Merge pull request #1792 from stefan-it/distilbert-for-token-classification 
DistilBERT for token classification
 2019-11-14 22:47:53 +01:00