Thomas Wolf
d43dc48b34
Merge branch 'master' into auto_models
2019-08-05 19:17:35 +02:00
thomwolf
0b524b0848
remove derived classes for now
2019-08-05 19:08:19 +02:00
thomwolf
13936a9621
update doc and tests
2019-08-05 18:48:16 +02:00
thomwolf
ed4e542260
adding tests
2019-08-05 18:14:07 +02:00
thomwolf
3a126e73dd
fix #950
2019-08-05 17:26:29 +02:00
thomwolf
b90e29d52c
working on automodels
2019-08-05 16:06:34 +02:00
thomwolf
328afb7097
cleaning up tokenizer tests structure (at last) - last remaining ppb refs
2019-08-05 14:08:56 +02:00
Thomas Wolf
0e918707dc
Merge pull request #907 from dhpollack/fix_convert_to_tf
...
Fix convert to tf
2019-08-05 12:55:04 +02:00
thomwolf
00132b7a7a
updating docs - adding few tests to tokenizers
2019-08-04 22:42:55 +02:00
thomwolf
009273dbdd
big doc update [WIP]
2019-08-04 12:14:57 +02:00
David Pollack
c90119e543
spelling mistake
2019-07-29 16:56:02 +02:00
thomwolf
bfbe52ec39
cleaning up example docstrings
2019-07-27 20:25:39 +02:00
thomwolf
4cc1bf81ee
typos
2019-07-27 12:08:21 +02:00
thomwolf
ac27548b25
fix unk_token test
2019-07-27 11:50:47 +02:00
thomwolf
c717d38573
dictionnary => dictionary
2019-07-26 23:30:48 +02:00
thomwolf
7b6e474c9a
fix #901
2019-07-26 21:26:44 +02:00
thomwolf
632d711411
fix #908
2019-07-26 21:14:37 +02:00
thomwolf
27b0f86d36
clean up pretrained
2019-07-26 17:09:21 +02:00
thomwolf
57e54ec070
add unk_token to gpt2
2019-07-26 17:09:07 +02:00
thomwolf
ac42049c08
add auto models and auto tokenizer
2019-07-26 17:08:59 +02:00
David Pollack
09ecf225e9
fixed the fix. tf session madness.
2019-07-26 15:20:44 +02:00
David Pollack
edfd965ac8
fix convert_to_tf
2019-07-26 14:13:46 +02:00
Thomas Wolf
6219ad7216
Merge pull request #888 from rococode/patch-1
...
Update docs for parameter rename
2019-07-25 15:01:22 +02:00
Joel Grus
ae152cec09
make save_pretrained work with added tokens
...
right now it's dumping the *decoder* when it should be dumping the *encoder*. this fixes that.
2019-07-24 16:54:48 -07:00
rococo // Ron
66b15f73f0
Update docs for parameter rename
...
OpenAIGPTLMHeadModel now accepts `labels` instead of `lm_labels`
2019-07-24 11:27:08 -07:00
Thomas Wolf
067923d326
Merge pull request #873 from huggingface/identity_replacement
...
Add nn.Identity replacement for old PyTorch
2019-07-23 18:16:35 +02:00
Thomas Wolf
368670ac31
Merge pull request #866 from xanlsh/master
...
Rework how PreTrainedModel.from_pretrained handles its arguments
2019-07-23 18:05:30 +02:00
thomwolf
1383c7b87a
Fix #869
2019-07-23 17:52:20 +02:00
Anish Moorthy
4fb56c7729
Remove unused *args parameter from PreTrainedConfig.from_pretrained
2019-07-23 10:43:01 -04:00
Anish Moorthy
e179c55490
Add docs for from_pretrained functions, rename return_unused_args
2019-07-23 10:43:01 -04:00
thomwolf
0740e63e49
updating schedules for state_dict saving
2019-07-23 15:57:18 +02:00
Thomas Wolf
c4bc66886d
Merge pull request #860 from Yiqing-Zhou/patch-1
...
read().splitlines() -> readlines()
2019-07-23 15:24:25 +02:00
Yiqing-Zhou
b1019d2a8e
token[-1] -> token.rstrip('\n')
2019-07-23 20:41:26 +08:00
thomwolf
0227b4a940
fix #827
2019-07-23 14:06:43 +02:00
Anish Moorthy
490ebbdcf7
Fix PretrainedModel.from_pretrained not passing cache_dir forward
2019-07-22 18:03:08 -04:00
Anish Moorthy
b8009cb0da
Make PreTrainedModel.from_pretrained pass unused arguments to model
2019-07-22 18:03:08 -04:00
Yiqing-Zhou
bef0c629ca
fix
...
Remove '\n' before adding token into vocab
2019-07-22 22:30:49 +08:00
Yiqing-Zhou
897d0841be
read().splitlines() -> readlines()
...
splitlines() does not work as what we expect here for bert-base-chinese because there is a '\u2028' (unicode line seperator) token in vocab file. Value of '\u2028'.splitlines() is ['', ''].
Perhaps we should use readlines() instead.
2019-07-22 20:49:09 +08:00
Minho Ryu
cd8980e1f4
import sys twice
2019-07-17 18:12:01 +09:00
thomwolf
5fe0b378d8
adding missing docstring fix #793
2019-07-16 21:35:53 +02:00
thomwolf
ed7549bb1a
release version 1.0
2019-07-16 16:10:58 +02:00
thomwolf
4acaa65068
model in evaluation mode by default after from_pretrained
2019-07-16 15:41:57 +02:00
thomwolf
f289e6cfe4
fix docstrings
2019-07-16 15:31:21 +02:00
thomwolf
9726b229cf
model name typo
2019-07-16 15:17:45 +02:00
thomwolf
1849aa7d39
update readme and pretrained model weight files
2019-07-16 15:11:29 +02:00
thomwolf
f31154cb9d
Merge branch 'xlnet'
2019-07-16 11:51:13 +02:00
thomwolf
1b35d05d4b
update conversion scripts and __main__
2019-07-16 09:41:55 +02:00
thomwolf
352e3ff998
added migration guide to readme
2019-07-16 09:03:49 +02:00
thomwolf
3b8b0e01bb
update readme
2019-07-16 00:12:55 +02:00
thomwolf
e691fc0963
update QA models tests + run_generation
2019-07-15 17:45:24 +02:00
Thomas Wolf