c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,180 Commits 171 Branches 49 Tags
c3ba6452377f085d0f59e15b97ac247bca24367e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Lysandre c3ba645237 Works for XLNet 2019-11-22 16:27:37 -05:00
LysandreJik a5a8a6175f Works for BERT 2019-11-22 16:27:31 -05:00
LysandreJik a7dafe2f41 Padding strategy (left and right) rather than boolean flag 2019-11-22 16:27:25 -05:00
LysandreJik 9f374c8252 encode and encode_plus handle attention masks and padding 2019-11-22 16:27:15 -05:00
Lysandre 72e506b22e wip 2019-11-22 16:26:00 -05:00
Lysandre ea52f82455 Moved some SQuAD logic to /data 2019-11-22 16:25:52 -05:00
Lysandre 454455c695 fix #1879 2019-11-20 09:42:48 -05:00
Lysandre be7f2aacce [CI][DOC] Don't rebuild if folder exists - Correct directory. 2019-11-14 14:54:44 -05:00
Lysandre 8f8d69716a [CI][DOC] Don't rebuild if folder exists. 2019-11-14 14:48:21 -05:00
Lysandre d7929899da Specify checkpoint in saved file for run_lm_finetuning.py 2019-11-14 10:49:00 -05:00
Lysandre a67e747889 Reorganized max_len warning 2019-11-14 10:30:22 -05:00
Lysandre e18f786cd5 Quickstart example showcasing past 2019-11-14 10:06:00 -05:00
Lysandre 74d0bcb6ff Fix special tokens addition in decoder 2019-11-12 15:27:57 -05:00
Lysandre d409aca326 Clarify the use of past in GPT2 and CTRL 2019-11-12 10:59:37 -05:00
Lysandre b5d330d118 Fix #1784 2019-11-11 10:15:14 -05:00
Lysandre d7d36181fd GPT-2 XL 2019-11-05 13:31:58 -05:00
LysandreandSantosh-Gupta 68f7064a3e Add model.train() line to ReadMe training example 
Co-Authored-By: Santosh-Gupta <San.Gupta.ML@gmail.com>
 2019-11-04 11:52:35 -05:00
Lysandre 1a2b40cb53 run_tf_glue MRPC evaluation only for MRPC 2019-10-31 18:00:51 -04:00
Lysandre beaf66b1f3 Remove break 2019-10-24 21:43:28 +00:00
Lysandre bab6ad01aa run_tf_glue works with all tasks 2019-10-24 21:41:45 +00:00
Lysandre b82bfbd0c3 Updated README to show all available documentation 2019-10-24 15:55:31 +00:00
Lysandre fbcc5ff9fb Change branch to master 2019-10-22 18:01:10 -04:00
Lysandre 69eba0ab19 Edit script path 2019-10-22 17:53:52 -04:00
Lysandre bc3e57d551 Multi version doc deployment 2019-10-22 17:51:30 -04:00
Lysandre 7d709e55ed Remove 2019-10-22 14:12:33 -04:00
Lysandre 44286b94d3 RoBERTa doesn't print a warning when no special tokens are passed. 2019-10-22 13:46:48 -04:00
Lysandre 1cfd974868 Option to benchmark only one of the two libraries 2019-10-22 13:32:23 -04:00
Lysandre 777faa8ae7 Fix #1597 2019-10-22 11:26:42 -04:00
LysandreJik 82f6abd98a Benchmark section added to the documentation 2019-10-18 17:27:10 -04:00
LysandreJik 7dd29ed2f1 Benchmarks example script 2019-10-18 10:53:04 -04:00
Lysandre a701c9b321 CTRL to tf automodels 2019-10-11 16:05:30 -04:00
Lysandre 3ddce1d74c Release: 2.1.1 2019-10-11 06:37:49 -04:00
Lysandre 639f4b7190 Don't save/load when on TPU 2019-10-10 19:17:25 +00:00
Lysandre d4e7934ac3 GLUE on TPU 2019-10-10 19:03:06 +00:00
LysandreJik 036483fae5 Temporary CTRL tokenizer fix 2019-10-09 16:33:15 -04:00
LysandreJik 9c2e0a4acf Release: 2.1.0 2019-10-09 12:14:03 -04:00
LysandreJik 7fe98d8c18 Update CTRL documentation 2019-10-09 12:12:36 -04:00
LysandreJik 89f86f9661 CTRL added to the documentation 2019-10-09 12:04:06 -04:00
LysandreJik e17ea08e24 Pycharm folder added to gitignore 2019-10-09 11:32:21 -04:00
LysandreJik 8fcc6507ce Multilingual 2019-10-07 15:02:42 -04:00
LysandreJik f3e0218fbb Correct device assignment in run_generation 2019-10-05 21:05:16 -04:00
LysandreJik 7bddb45a6f Decode documentaton 2019-10-04 14:27:38 -04:00
LysandreJik ecc4f1bdfa XLM use_lang_embedding flag in run_generation 2019-10-03 17:42:16 -04:00
LysandreJik c2c2ca0fdb Added XLM to run_generation, with prompt language selection. 2019-10-03 17:18:48 -04:00
LysandreJik aebd83230f Update naming + remove f string in run_lm_finetuning example 2019-10-03 11:31:36 -04:00
LysandreJik 651bfb7ad5 always_truncate by default 2019-10-03 11:31:36 -04:00
LysandreJik 5ed50a93fb LM finetuning won't mask special tokens anymore 2019-10-03 11:31:36 -04:00
LysandreJik cc412edd42 Supports already existing special tokens 2019-10-03 11:31:36 -04:00
LysandreJik 2f259b228e Sequence IDS 2019-10-03 11:31:36 -04:00
LysandreJik 7c789c337d Always truncate argument in the encode method 2019-10-03 11:31:36 -04:00
LysandreJik ebb32261b1 fix #1401 2019-10-02 17:52:56 -04:00
LysandreJik ecfddc6034 Update RoBERTa and GPT-2 Tokenizer documentation (fix #1343) 2019-09-26 16:49:03 -04:00
LysandreJik 93f0c5fc72 Repository link in the documentation 2019-09-26 11:45:00 -04:00
LysandreJik 7e957237e4 [Doc] XLM + Torch in documentation 2019-09-26 10:08:56 -04:00
LysandreJik 302a4813a5 Doc building requirements [TF2] 2019-09-26 09:57:30 -04:00
LysandreJik a3e0dbba95 Doc building requirements [TF] 2019-09-26 09:51:14 -04:00
LysandreJik 4094958df2 Doc building requirements 2019-09-26 08:50:55 -04:00
LysandreJik 7d8b395afa Doc building requirements 2019-09-26 08:49:31 -04:00
LysandreJik 927904bc91 [doc] pytorch_transformers -> transformers 2019-09-26 08:47:15 -04:00
LysandreJik 294edfd83d Release version in documentation 2019-09-26 08:16:12 -04:00
LysandreJik de5e4864cb Documentation 2019-09-26 08:04:54 -04:00
LysandreJik 8349d75773 Various small doc fixes 2019-09-26 07:45:40 -04:00
LysandreJik fb056494e5 Example usage 2019-09-26 07:45:40 -04:00
LysandreJik 36f592cc82 Updated doc for InputExample and InputFeatures 2019-09-26 07:45:40 -04:00
LysandreJik ad4a393e2e Changed processor documentation architecture. Added documentation for GLUE 2019-09-26 07:45:40 -04:00
LysandreJik c4ac7a76db GLUE processors 2019-09-26 07:45:40 -04:00
LysandreJik 4acd87ff4e TF models added to documentation 2019-09-26 07:45:40 -04:00
LysandreJik cf5c5c9e1c Documentation 2019-09-26 07:43:13 -04:00
LysandreJik 1761d2091a Check to see if the models have the same results when in eval mode (pt) or when training=False (tf) 2019-09-24 14:59:10 -04:00
LysandreJik 0b82e3d0d9 Relative imports 2019-09-24 09:52:25 -04:00
LysandreJik f09e5ecef0 [Proposal] GLUE processors included in library 2019-09-24 09:47:34 -04:00
LysandreJik 72402d1acd Fixed DistilBERT tokenizer 2019-09-24 09:41:14 -04:00
LysandreJik d340e2329e create_mask_from_sequences -> create_token_type_ids_from_sequences 2019-09-24 09:09:28 -04:00
LysandreJik c832f43a4d output_token_type -> token_type_ids 2019-09-24 07:21:38 -04:00
LysandreJik 3927d7756c Updated the GLUE pre-processing method 2019-09-24 07:15:11 -04:00
LysandreJik 0ea82b246f Updated tests 2019-09-24 07:10:09 -04:00
LysandreJik 9d44236f70 Updated DistilBERT 2019-09-24 07:03:24 -04:00
LysandreJik ab984a8b72 Python 2 compatibility 2019-09-19 15:01:33 +02:00
LysandreJik 3df208c93a Tokenizer accepts token list as well as string 2019-09-19 14:47:52 +02:00
LysandreJik 66ea76b8a9 prepare_for_model and prepare_pair_for_model methods. Added an option to select which sequence will be truncated. 2019-09-19 13:50:51 +02:00
LysandreJik 60414f31a9 GLUE updated with new methods 2019-09-19 10:55:06 +02:00
LysandreJik baa74326ab Stride + tests + small fixes 2019-09-19 10:55:06 +02:00
LysandreJik c10c7d59e7 Mask computing in standalone method. Tests. 2019-09-19 10:55:06 +02:00
LysandreJik bf503158c5 Sentence -> Sequence. Removed output_mask from the special token addition methods. 2019-09-19 10:55:06 +02:00
LysandreJik 8cba057260 Doc + remove artefacts 2019-09-19 10:55:06 +02:00
LysandreJik 6393261e41 encode + encode_plus tests modified 2019-09-19 10:55:06 +02:00
LysandreJik dcc9bb3252 Modified encode to return only lists. Added a more complete encode_plus method 2019-09-19 10:55:06 +02:00
LysandreJik af23b626c8 Max encoding length + corresponding tests 2019-09-19 10:55:06 +02:00
LysandreJik c4d4f3ec8c Updated DistilBERT test to reflect the sequence encoding 2019-09-19 10:55:06 +02:00
LysandreJik d572d7027b Number of added tokens calculator 2019-09-19 10:55:06 +02:00
LysandreJik de8e14b6c0 Added DistilBERT to run_squad script 2019-09-19 10:55:06 +02:00
LysandreJik 88368c2a16 Added DistilBERT to run_lm_finetuning 2019-09-19 10:55:06 +02:00
LysandreJik 2d8ec5a684 Changed warning to be more explicit 
Co-authored by: julien_c <chaumond@gmail.com>
 2019-09-19 10:55:06 +02:00
LysandreJik 75635072e1 Updated GLUE script to add DistilBERT. Cleaned up unused args in the utils file. 2019-09-19 10:55:06 +02:00
LysandreJik 92a9976e91 Distilbert sequence builder w/ mask 2019-09-19 10:55:06 +02:00
LysandreJik 59057abe52 typo 2019-09-19 10:55:06 +02:00
LysandreJik bac332fec0 Updated the GLUE data processor. Corrections to RoBERTa and XLNet. 2019-09-19 10:55:06 +02:00
LysandreJik c3df2136e1 Added binary masking tests 2019-09-19 10:55:06 +02:00
LysandreJik e391d4735e Tokenizers' encode function can output binary masks 2019-09-19 10:55:06 +02:00
LysandreJik 11ac4b9555 [CI] Symbolic link for documentation 2019-09-11 10:13:44 +02:00