c-my/transformers
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0ce2f496dc4578e0dc2be8ff02783600d23dc635
transformers/notebooks/Comparing TF and PT models.ipynb
T
thomwolf e6646751ac update notebooks
2018-11-05 15:02:50 +01:00

51 KiB
Raw Blame History

Comparing TensorFlow (original) and PyTorch models

You can use this small notebook to check the conversion of the model's weights from the TensorFlow model to the PyTorch model. In the following, we compare the weights of the last layer on a simple example (in input.txt) but both models returns all the hidden layers so you can check every stage of the model.

To run this notebook, follow these instructions:

  • make sure that your Python environment has both TensorFlow and PyTorch installed,
  • download the original TensorFlow implementation,
  • download a pre-trained TensorFlow model as indicaded in the TensorFlow implementation readme,
  • run the script convert_tf_checkpoint_to_pytorch.py as indicated in the README to convert the pre-trained TensorFlow model to PyTorch.

If needed change the relative paths indicated in this notebook (at the beggining of Sections 1 and 2) to point to the relevent models and code.

In [1]:
import os
os.chdir('../')

1/ TensorFlow code

In [2]:
original_tf_inplem_dir = "./tensorflow_code/"
model_dir = "../google_models/uncased_L-12_H-768_A-12/"

vocab_file = model_dir + "vocab.txt"
bert_config_file = model_dir + "bert_config.json"
init_checkpoint = model_dir + "bert_model.ckpt"

input_file = "./samples/input.txt"
max_seq_length = 128
In [3]:
import importlib.util
import sys

spec = importlib.util.spec_from_file_location('*', original_tf_inplem_dir + '/extract_features.py')
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
sys.modules['extract_features_tensorflow'] = module

from extract_features_tensorflow import *
In [4]:
layer_indexes = list(range(12))
bert_config = modeling.BertConfig.from_json_file(bert_config_file)
tokenizer = tokenization.FullTokenizer(
    vocab_file=vocab_file, do_lower_case=True)
examples = read_examples(input_file)

features = convert_examples_to_features(
    examples=examples, seq_length=max_seq_length, tokenizer=tokenizer)
unique_id_to_feature = {}
for feature in features:
    unique_id_to_feature[feature.unique_id] = feature
INFO:tensorflow:*** Example ***
INFO:tensorflow:unique_id: 0
INFO:tensorflow:tokens: [CLS] who was jim henson ? [SEP] jim henson was a puppet ##eer [SEP]
INFO:tensorflow:input_ids: 101 2040 2001 3958 27227 1029 102 3958 27227 2001 1037 13997 11510 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
INFO:tensorflow:input_type_ids: 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
In [5]:
is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2
run_config = tf.contrib.tpu.RunConfig(
    master=None,
    tpu_config=tf.contrib.tpu.TPUConfig(
        num_shards=1,
        per_host_input_for_training=is_per_host))

model_fn = model_fn_builder(
    bert_config=bert_config,
    init_checkpoint=init_checkpoint,
    layer_indexes=layer_indexes,
    use_tpu=False,
    use_one_hot_embeddings=False)

# If TPU is not available, this will fall back to normal Estimator on CPU
# or GPU.
estimator = tf.contrib.tpu.TPUEstimator(
    use_tpu=False,
    model_fn=model_fn,
    config=run_config,
    predict_batch_size=1)

input_fn = input_fn_builder(
    features=features, seq_length=max_seq_length)
WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x12839dbf8>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:Using temporary folder as model directory: /var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmpdbx_h23u
INFO:tensorflow:Using config: {'_model_dir': '/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmpdbx_h23u', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x12b3e1c18>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=2, num_shards=1, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': None}
WARNING:tensorflow:Setting TPUConfig.num_shards==1 is an unsupported behavior. Please fix as soon as possible (leaving num_shards as None.
INFO:tensorflow:_TPUContext: eval_on_tpu True
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
In [6]:
tensorflow_all_out = []
for result in estimator.predict(input_fn, yield_single_examples=True):
    unique_id = int(result["unique_id"])
    feature = unique_id_to_feature[unique_id]
    output_json = collections.OrderedDict()
    output_json["linex_index"] = unique_id
    tensorflow_all_out_features = []
    # for (i, token) in enumerate(feature.tokens):
    all_layers = []
    for (j, layer_index) in enumerate(layer_indexes):
        print("extracting layer {}".format(j))
        layer_output = result["layer_output_%d" % j]
        layers = collections.OrderedDict()
        layers["index"] = layer_index
        layers["values"] = layer_output
        all_layers.append(layers)
    tensorflow_out_features = collections.OrderedDict()
    tensorflow_out_features["layers"] = all_layers
    tensorflow_all_out_features.append(tensorflow_out_features)

    output_json["features"] = tensorflow_all_out_features
    tensorflow_all_out.append(output_json)
INFO:tensorflow:Could not find trained model in model_dir: /var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmpdbx_h23u, running initialization to predict.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Running infer on CPU
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
extracting layer 0
extracting layer 1
extracting layer 2
extracting layer 3
extracting layer 4
extracting layer 5
extracting layer 6
extracting layer 7
extracting layer 8
extracting layer 9
extracting layer 10
extracting layer 11
INFO:tensorflow:prediction_loop marked as finished
INFO:tensorflow:prediction_loop marked as finished
In [7]:
print(len(tensorflow_all_out))
print(len(tensorflow_all_out[0]))
print(tensorflow_all_out[0].keys())
print("number of tokens", len(tensorflow_all_out[0]['features']))
print("number of layers", len(tensorflow_all_out[0]['features'][0]['layers']))
tensorflow_all_out[0]['features'][0]['layers'][0]['values'].shape
Out [7]:
1
2
odict_keys(['linex_index', 'features'])
number of tokens 1
number of layers 12

(128, 768)
In [8]:
tensorflow_outputs = list(tensorflow_all_out[0]['features'][0]['layers'][t]['values'] for t in layer_indexes)

2/ PyTorch code

In [9]:
import extract_features
from extract_features import *
In [10]:
init_checkpoint_pt = "../google_models/uncased_L-12_H-768_A-12/pytorch_model.bin"
In [11]:
device = torch.device("cpu")
model = extract_features.BertModel(bert_config)
model.load_state_dict(torch.load(init_checkpoint_pt, map_location='cpu'))
model.to(device)
Out [11]:
BertModel(
  (embeddings): BERTEmbeddings(
    (word_embeddings): Embedding(30522, 768)
    (position_embeddings): Embedding(512, 768)
    (token_type_embeddings): Embedding(2, 768)
    (LayerNorm): BERTLayerNorm()
    (dropout): Dropout(p=0.1)
  )
  (encoder): BERTEncoder(
    (layer): ModuleList(
      (0): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (1): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (2): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (3): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (4): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (5): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (6): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (7): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (8): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (9): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (10): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (11): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
    )
  )
  (pooler): BERTPooler(
    (dense): Linear(in_features=768, out_features=768, bias=True)
    (activation): Tanh()
  )
)
In [12]:
all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
all_input_mask = torch.tensor([f.input_mask for f in features], dtype=torch.long)
all_input_type_ids = torch.tensor([f.input_type_ids for f in features], dtype=torch.long)
all_example_index = torch.arange(all_input_ids.size(0), dtype=torch.long)

eval_data = TensorDataset(all_input_ids, all_input_mask, all_input_type_ids, all_example_index)
eval_sampler = SequentialSampler(eval_data)
eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=1)

model.eval()
Out [12]:
BertModel(
  (embeddings): BERTEmbeddings(
    (word_embeddings): Embedding(30522, 768)
    (position_embeddings): Embedding(512, 768)
    (token_type_embeddings): Embedding(2, 768)
    (LayerNorm): BERTLayerNorm()
    (dropout): Dropout(p=0.1)
  )
  (encoder): BERTEncoder(
    (layer): ModuleList(
      (0): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (1): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (2): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (3): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (4): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (5): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (6): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (7): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (8): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (9): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (10): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
      (11): BERTLayer(
        (attention): BERTAttention(
          (self): BERTSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1)
          )
          (output): BERTSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): BERTLayerNorm()
            (dropout): Dropout(p=0.1)
          )
        )
        (intermediate): BERTIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BERTOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): BERTLayerNorm()
          (dropout): Dropout(p=0.1)
        )
      )
    )
  )
  (pooler): BERTPooler(
    (dense): Linear(in_features=768, out_features=768, bias=True)
    (activation): Tanh()
  )
)
In [13]:
layer_indexes = list(range(12))

pytorch_all_out = []
for input_ids, input_mask, input_type_ids, example_indices in eval_dataloader:
    print(input_ids)
    print(input_mask)
    print(example_indices)
    input_ids = input_ids.to(device)
    input_mask = input_mask.to(device)

    all_encoder_layers, _ = model(input_ids, token_type_ids=input_type_ids, attention_mask=input_mask)

    for b, example_index in enumerate(example_indices):
        feature = features[example_index.item()]
        unique_id = int(feature.unique_id)
        # feature = unique_id_to_feature[unique_id]
        output_json = collections.OrderedDict()
        output_json["linex_index"] = unique_id
        all_out_features = []
        # for (i, token) in enumerate(feature.tokens):
        all_layers = []
        for (j, layer_index) in enumerate(layer_indexes):
            print("layer", j, layer_index)
            layer_output = all_encoder_layers[int(layer_index)].detach().cpu().numpy()
            layer_output = layer_output[b]
            layers = collections.OrderedDict()
            layers["index"] = layer_index
            layer_output = layer_output
            layers["values"] = layer_output if not isinstance(layer_output, (int, float)) else [layer_output]
            all_layers.append(layers)

            out_features = collections.OrderedDict()
            out_features["layers"] = all_layers
            all_out_features.append(out_features)
        output_json["features"] = all_out_features
        pytorch_all_out.append(output_json)
tensor([[  101,  2040,  2001,  3958, 27227,  1029,   102,  3958, 27227,  2001,
          1037, 13997, 11510,   102,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0]])
tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0]])
tensor([0])
layer 0 0
layer 1 1
layer 2 2
layer 3 3
layer 4 4
layer 5 5
layer 6 6
layer 7 7
layer 8 8
layer 9 9
layer 10 10
layer 11 11
In [14]:
print(len(pytorch_all_out))
print(len(pytorch_all_out[0]))
print(pytorch_all_out[0].keys())
print("number of tokens", len(pytorch_all_out))
print("number of layers", len(pytorch_all_out[0]['features'][0]['layers']))
print("hidden_size", len(pytorch_all_out[0]['features'][0]['layers'][0]['values']))
pytorch_all_out[0]['features'][0]['layers'][0]['values'].shape
Out [14]:
1
2
odict_keys(['linex_index', 'features'])
number of tokens 1
number of layers 12
hidden_size 128

(128, 768)
In [15]:
pytorch_outputs = list(pytorch_all_out[0]['features'][0]['layers'][t]['values'] for t in layer_indexes)
print(pytorch_outputs[0].shape)
print(pytorch_outputs[1].shape)
(128, 768)
(128, 768)
In [16]:
print(tensorflow_outputs[0].shape)
print(tensorflow_outputs[1].shape)
(128, 768)
(128, 768)

3/ Comparing the standard deviation on the last layer of both models

In [17]:
import numpy as np
In [18]:
print('shape tensorflow layer, shape pytorch layer, standard deviation')
print('\n'.join(list(str((np.array(tensorflow_outputs[i]).shape,
                          np.array(pytorch_outputs[i]).shape, 
                          np.sqrt(np.mean((np.array(tensorflow_outputs[i]) - np.array(pytorch_outputs[i]))**2.0)))) for i in range(12))))
shape tensorflow layer, shape pytorch layer, standard deviation
((128, 768), (128, 768), 1.5258875e-07)
((128, 768), (128, 768), 2.342731e-07)
((128, 768), (128, 768), 2.801949e-07)
((128, 768), (128, 768), 3.5904986e-07)
((128, 768), (128, 768), 4.2842768e-07)
((128, 768), (128, 768), 5.127951e-07)
((128, 768), (128, 768), 6.14668e-07)
((128, 768), (128, 768), 7.063922e-07)
((128, 768), (128, 768), 7.906173e-07)
((128, 768), (128, 768), 8.475192e-07)
((128, 768), (128, 768), 8.975489e-07)
((128, 768), (128, 768), 4.1671223e-07)
In [ ]:
Reference in new issue View Git Blame Copy Permalink