c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

LayerNorm initialization

Browse Source 
The LayerNorm gamma and beta should be initialized by .fill_(1.0) and .zero_().

reference links:

https://github.com/tensorflow/tensorflow/blob/989e78c412a7e0f5361d4d7dfdfb230c8136e749/tensorflow/contrib/layers/python/layers/layers.py#L2298

https://github.com/tensorflow/tensorflow/blob/989e78c412a7e0f5361d4d7dfdfb230c8136e749/tensorflow/contrib/layers/python/layers/layers.py#L2308
This commit is contained in:
Li Dong authored and GitHub committed 2019-01-07 15:51:33 +08:00
1 parent 8da280ebbe
commit d0d9b384f2
1 file changed
+2 -2
Download Patch File Download Diff File Expand all files Collapse all files
pytorch_pretrained_bert/modeling.py
+2 -2
View File
@@ -439,8 +439,8 @@ class PreTrainedBertModel(nn.Module):
# cf https://github.com/pytorch/pytorch/pull/5617
module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
elif isinstance(module, BertLayerNorm):
module.bias.data.normal_(mean=0.0, std=self.config.initializer_range)
module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
module.bias.data.zero_()
module.weight.data.fill_(1.0)
if isinstance(module, nn.Linear) and module.bias is not None:
module.bias.data.zero_()