Rémi Louf
3cf2020c6b
change kwargs processing
2019-10-30 16:27:51 +01:00
Rémi Louf
a88a0e4413
add tests to encoder-decoder model
2019-10-30 16:06:29 +01:00
Rémi Louf
3f07cd419c
update test on Bert to include decoder mode
2019-10-30 15:09:53 +01:00
Rémi Louf
070507df1f
format utils for summarization
2019-10-30 11:24:12 +01:00
Rémi Louf
da10de8466
fix bug with padding mask + add corresponding test
2019-10-30 11:19:58 +01:00
Rémi Louf
3b0d2fa30e
rename seq2seq to encoder_decoder
2019-10-30 10:54:46 +01:00
Rémi Louf
9c1bdb5b61
revert renaming of lm_labels to ltr_lm_labels
2019-10-30 10:43:13 +01:00
Rémi Louf
098a89f312
update docstrings; rename lm_labels to more explicit ltr_lm_labels
2019-10-29 20:08:03 +01:00
Rémi Louf
dfce409691
resolve PR comments
2019-10-29 17:10:20 +01:00
Rémi Louf
4c3ac4a7d8
here's one big commit
2019-10-28 10:49:50 +01:00
Rémi Louf
932543f77e
fix test of truncation function
2019-10-28 10:49:49 +01:00
Rémi Louf
a67413ccc8
extend works in-place
2019-10-28 10:49:49 +01:00
Rémi Louf
cb26b035c6
remove potential UndefinedError
2019-10-28 10:49:49 +01:00
Rémi Louf
b915ba9dfe
pad sequence with 0, mask with -1
2019-10-28 10:49:49 +01:00
Rémi Louf
dc580dd4c7
add lm_labels for the LM cross-entropy
2019-10-28 10:49:49 +01:00
Rémi Louf
f873a3edb2
the decoder attends to the output of the encoder stack (last layer)
2019-10-28 10:49:00 +01:00
Rémi Louf
578d23e061
add training pipeline (formatting temporary)
2019-10-17 14:02:27 +02:00
Rémi Louf
47a06d88a0
use two different tokenizers for storyand summary
2019-10-17 13:04:26 +02:00
Rémi Louf
bfb9b540d4
add Model2Model to __init__
2019-10-17 12:59:51 +02:00
Rémi Louf
c1bc709c35
correct the truncation and padding of dataset
2019-10-17 10:41:53 +02:00
Rémi Louf
87d60b6e19
reword explanation of encoder_attention_mask
2019-10-17 10:18:19 +02:00
Rémi Louf
638fe7f5a4
correct composition of padding and causal masks
2019-10-17 10:13:07 +02:00
Rémi Louf
4e0f24348f
document the MLM modification + raise exception on MLM training with encoder-decoder
2019-10-17 09:41:53 +02:00
Rémi Louf
624a5644cc
revert black formatting to conform with lib style
2019-10-17 09:27:56 +02:00
Rémi Louf
9b71fc9a18
tying weights is going to be a clusterfuck
2019-10-16 21:31:38 +02:00
Rémi Louf
95ec1d08be
separate inputs into encoder & decoder inputs
2019-10-16 20:55:42 +02:00
Rémi Louf
e4e0ee14bd
add separator between data import and train
2019-10-16 20:05:32 +02:00
Rémi Louf
a424892fab
correct syntax error: dim() and not dims()
2019-10-16 18:24:32 +02:00
Rémi Louf
33c01368b1
remove Bert2Rnd test
2019-10-16 18:13:05 +02:00
Rémi Louf
260ac7d9a8
wip commit, switching computers
2019-10-15 12:24:35 +02:00
Rémi Louf
fe25eefc15
add instructions to fetch the dataset
2019-10-14 20:45:39 +02:00
Rémi Louf
412793275d
delegate the padding with special tokens to the tokenizer
2019-10-14 20:45:16 +02:00
Rémi Louf
447fffb21f
process the raw CNN/Daily Mail dataset
...
the data provided by Li Dong et al. were already tokenized, which means
that they are not compatible with all the models in the library. We
thus process the raw data directly and tokenize them using the models'
tokenizers.
2019-10-14 18:12:20 +02:00
Rémi Louf
67d10960ae
load and prepare CNN/Daily Mail data
...
We write a function to load an preprocess the CNN/Daily Mail dataset as
provided by Li Dong et al. The issue is that this dataset has already
been tokenized by the authors, so we actually need to find the original,
plain-text dataset if we want to apply it to all models.
2019-10-14 14:11:20 +02:00
Rémi Louf
b3261e7ace
read parameters from CLI, load model & tokenizer
2019-10-11 18:40:38 +02:00
Rémi Louf
d889e0b71b
add base for seq2seq finetuning
2019-10-11 17:36:12 +02:00
Rémi Louf
f8e98d6779
load pretrained embeddings in Bert decoder
...
In Rothe et al.'s "Leveraging Pre-trained Checkpoints for Sequence
Generation Tasks", Bert2Bert is initialized with pre-trained weights for
the encoder, and only pre-trained embeddings for the decoder. The
current version of the code completely randomizes the weights of the
decoder.
We write a custom function to initiliaze the weights of the decoder; we
first initialize the decoder with the weights and then randomize
everything but the embeddings.
2019-10-11 16:48:11 +02:00
Rémi Louf
1e68c28670
add test for initialization of Bert2Rnd
2019-10-10 18:07:11 +02:00
Rémi Louf
fa218e648a
fix syntax errors
2019-10-10 15:16:07 +02:00
Rémi Louf
3e1cd8241e
fix stupid (re)naming issue
2019-10-10 14:18:20 +02:00
Rémi Louf
81ee29ee8d
remove the staticmethod used to load the config
2019-10-10 14:13:37 +02:00
Rémi Louf
d7092d592c
rename the attributes in the Bert Layer
...
Since the preloading of weights relies on the name of the class's
attributes changing the namespace breaks loading pretrained weights on
Bert and all related models. I reverted `self_attention` to `attention`
and us `crossattention` for the decoder instead.
2019-10-10 12:51:14 +02:00
Rémi Louf
51261167b4
prune both attention and self-attention heads
2019-10-10 12:17:22 +02:00
Rémi Louf
17177e7379
add is_decoder as an attribute to Config class
2019-10-10 12:03:58 +02:00
Rémi Louf
df85a0ff0b
replace double quotes with simple quotes
2019-10-10 11:38:26 +02:00
Rémi Louf
9ca788b2e8
merge the two Bert layers classes
2019-10-10 11:33:28 +02:00
Rémi Louf
edfc8f8225
Remove and do the branching in
2019-10-10 10:17:27 +02:00
Rémi Louf
09cfd12235
remove and do the branching in
2019-10-10 10:15:27 +02:00
Rémi Louf
877ef2c6ca
override from_pretrained in Bert2Rnd
...
In the seq2seq model we need to both load pretrained weights in the
encoder and initialize the decoder randomly. Because the
`from_pretrained` method defined in the base class relies on module
names to assign weights, it would also initialize the decoder with
pretrained weights. To avoid this we override the method to only
initialize the encoder with pretrained weights.
2019-10-10 10:02:18 +02:00
Rémi Louf
851ef592c5
add comment on recursive weights loading
2019-10-10 10:02:03 +02:00
Rémi Louf
770b15b58c
rename class in __init__
2019-10-08 17:32:28 +02:00
Rémi Louf
f4d41fe33e
Merge pull request #1448 from huggingface/contributing
...
add contribution guidelines
2019-10-08 16:55:34 +02:00
Rémi Louf
6e3e1c959e
Merge pull request #1447 from huggingface/dev-requirements
...
Provide requirements.txt for development dependencies
2019-10-07 18:49:26 +02:00
Rémi Louf