thomwolf
c9591f6fac
updated models input format + tests
2019-09-23 22:08:08 +02:00
Julien Chaumond
798da627eb
Fix TFBert tests in Python 3.5
2019-09-23 12:06:10 -04:00
thomwolf
c014d1f0c6
fix the skipping
2019-09-23 16:39:57 +02:00
thomwolf
0b22e47a40
skipping pretrained TF model tests for now
2019-09-23 16:38:03 +02:00
thomwolf
830d212be7
test circleCI h5py version
2019-09-23 16:26:06 +02:00
Thomas Wolf
7c0f2d0a6a
Merge pull request #1294 from sshleifer/delete-n-special-doc
...
Delete n_special reference in docstring
2019-09-23 14:54:55 +01:00
thomwolf
a31e591d27
fix XLM tests
2019-09-23 15:54:10 +02:00
thomwolf
447de34dde
tests for distilbert and roberta
2019-09-23 15:38:29 +02:00
Santiago Castro
98dd19b96b
Remove unnecessary use of FusedLayerNorm
2019-09-22 20:31:36 -04:00
Lorenzo Ampil
4b543c3007
Add option to use a 'stop token' which will be used to truncate the output text to everything till right before the 'stop token'
2019-09-22 21:38:38 +08:00
thomwolf
68a3e0223a
roberta and distilbert
2019-09-20 23:14:51 +02:00
Maxpa1n
a2d4950f5c
fix annotation
2019-09-20 10:59:35 -04:00
VictorSanh
9f995b99d4
minor fixes
2019-09-19 21:36:06 +00:00
VictorSanh
3fe5c8e8a8
update bert-base-uncased rslts
2019-09-19 19:34:22 +00:00
VictorSanh
354944e607
[distillation] big update w/ new weights
2019-09-19 19:25:21 +00:00
danai-antoniou
2e6797cc7d
Added valuerror for duplicate added tokens
2019-09-19 15:40:42 +01:00
LysandreJik
ab984a8b72
Python 2 compatibility
2019-09-19 15:01:33 +02:00
LysandreJik
3df208c93a
Tokenizer accepts token list as well as string
2019-09-19 14:47:52 +02:00
LysandreJik
66ea76b8a9
prepare_for_model and prepare_pair_for_model methods. Added an option to select which sequence will be truncated.
2019-09-19 13:50:51 +02:00
LysandreJik
60414f31a9
GLUE updated with new methods
2019-09-19 10:55:06 +02:00
LysandreJik
baa74326ab
Stride + tests + small fixes
2019-09-19 10:55:06 +02:00
LysandreJik
c10c7d59e7
Mask computing in standalone method. Tests.
2019-09-19 10:55:06 +02:00
LysandreJik
bf503158c5
Sentence -> Sequence. Removed output_mask from the special token addition methods.
2019-09-19 10:55:06 +02:00
LysandreJik
8cba057260
Doc + remove artefacts
2019-09-19 10:55:06 +02:00
LysandreJik
6393261e41
encode + encode_plus tests modified
2019-09-19 10:55:06 +02:00
LysandreJik
dcc9bb3252
Modified encode to return only lists. Added a more complete encode_plus method
2019-09-19 10:55:06 +02:00
LysandreJik
af23b626c8
Max encoding length + corresponding tests
2019-09-19 10:55:06 +02:00
LysandreJik
c4d4f3ec8c
Updated DistilBERT test to reflect the sequence encoding
2019-09-19 10:55:06 +02:00
LysandreJik
d572d7027b
Number of added tokens calculator
2019-09-19 10:55:06 +02:00
LysandreJik
de8e14b6c0
Added DistilBERT to run_squad script
2019-09-19 10:55:06 +02:00
LysandreJik
88368c2a16
Added DistilBERT to run_lm_finetuning
2019-09-19 10:55:06 +02:00
LysandreJik
2d8ec5a684
Changed warning to be more explicit
...
Co-authored by: julien_c <chaumond@gmail.com >
2019-09-19 10:55:06 +02:00
LysandreJik
75635072e1
Updated GLUE script to add DistilBERT. Cleaned up unused args in the utils file.
2019-09-19 10:55:06 +02:00
LysandreJik
92a9976e91
Distilbert sequence builder w/ mask
2019-09-19 10:55:06 +02:00
LysandreJik
59057abe52
typo
2019-09-19 10:55:06 +02:00
LysandreJik
bac332fec0
Updated the GLUE data processor. Corrections to RoBERTa and XLNet.
2019-09-19 10:55:06 +02:00
LysandreJik
c3df2136e1
Added binary masking tests
2019-09-19 10:55:06 +02:00
LysandreJik
e391d4735e
Tokenizers' encode function can output binary masks
2019-09-19 10:55:06 +02:00
sshleifer
119610b5c5
Merge branch 'master' into delete-n-special-doc
2019-09-19 01:35:01 -07:00
sshleifer
08e4ad5eea
Remove documentation for unused kwarg
2019-09-18 16:35:01 -07:00
Erik Chan
f0340eccf9
Typo
...
Typo
2019-09-18 13:42:11 -07:00
Thomas Wolf
0d1dad6d53
Merge pull request #1004 from erenup/master
...
Refactoring old run_swag.py
2019-09-18 21:42:51 +02:00
erenup
8960988f35
fixed to find best dev acc
2019-09-19 01:10:05 +08:00
erenup
b57bfb5fa0
Merge pull request #3 from erenup/run_multiple_choice_merge
...
Run multiple choice merge
2019-09-18 21:45:04 +08:00
erenup
46ffc28329
Merge branch 'master' into run_multiple_choice_merge
...
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
2019-09-18 21:43:46 +08:00
Simon Layton
ec94f4e0f8
Fix fp16 masking in PoolerEndLogits
...
Necessary to run xlnet (at least in squad) with `--fp16 --fp16_opt_level="O2"`, otherwise loss is immediately `NaN` and fine-tuning cannot proceed.
2019-09-18 09:30:58 -04:00
erenup
15143fbad6
move run_multiple_choice.py and utils_multiple_choice.py to examples
2019-09-18 21:18:46 +08:00
erenup
3cd6289758
Merge remote-tracking branch 'huggingface/master' into run_multiple_choice_merge
...
# Conflicts:
# examples/contrib/run_swag.py
2019-09-18 21:16:59 +08:00
erenup
36362cf086
move schedule.step after optimizer.step
2019-09-18 21:13:40 +08:00
thomwolf
3a527fa820
OpenAI GPT tests ok
2019-09-18 14:15:48 +02:00
thomwolf