First commit: adding all files from tapas_v3
This commit is contained in:
Rogge Niels
@@ -222,6 +222,7 @@ Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.
|
||||
ultilingual BERT into [DistilmBERT](https://github.com/huggingface/transformers/tree/master/examples/distillation) and a German version of DistilBERT.
|
||||
1. **[SqueezeBert](https://huggingface.co/transformers/model_doc/squeezebert.html)** released with the paper [SqueezeBERT: What can computer vision teach NLP about efficient neural networks?](https://arxiv.org/abs/2006.11316) by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, and Kurt W. Keutzer.
|
||||
1. **[T5](https://huggingface.co/transformers/model_doc/t5.html)** (from Google AI) released with the paper [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683) by Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.
|
||||
1. **[TAPAS](https://huggingface.co/transformers/master/model_doc/tapas.html)** released with the paper [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://arxiv.org/abs/2004.02349) by Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno and Julian Martin Eisenschlos.
|
||||
1. **[Transformer-XL](https://huggingface.co/transformers/model_doc/transformerxl.html)** (from Google/CMU) released with the paper [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860) by Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.
|
||||
1. **[XLM](https://huggingface.co/transformers/model_doc/xlm.html)** (from Facebook) released together with the paper [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample and Alexis Conneau.
|
||||
1. **[XLM-ProphetNet](https://huggingface.co/transformers/model_doc/xlmprophetnet.html)** (from Microsoft Research) released with the paper [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://arxiv.org/abs/2001.04063) by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang and Ming Zhou.
|
||||
|
||||
@@ -176,19 +176,22 @@ and conversion utilities for the following models:
|
||||
30. :doc:`T5 <model_doc/t5>` (from Google AI) released with the paper `Exploring the Limits of Transfer Learning with a
|
||||
Unified Text-to-Text Transformer <https://arxiv.org/abs/1910.10683>`__ by Colin Raffel and Noam Shazeer and Adam
|
||||
Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.
|
||||
31. :doc:`Transformer-XL <model_doc/transformerxl>` (from Google/CMU) released with the paper `Transformer-XL:
|
||||
31. :doc:`TAPAS <model_doc/tapas>` (from Google AI) released with the paper `TAPAS: Weakly Supervised Table Parsing via
|
||||
Pre-training <https://arxiv.org/abs/2004.02349>`__ by Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller,
|
||||
Francesco Piccinno and Julian Martin Eisenschlos.
|
||||
32. :doc:`Transformer-XL <model_doc/transformerxl>` (from Google/CMU) released with the paper `Transformer-XL:
|
||||
Attentive Language Models Beyond a Fixed-Length Context <https://arxiv.org/abs/1901.02860>`__ by Zihang Dai*,
|
||||
Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.
|
||||
32. :doc:`XLM <model_doc/xlm>` (from Facebook) released together with the paper `Cross-lingual Language Model
|
||||
33. :doc:`XLM <model_doc/xlm>` (from Facebook) released together with the paper `Cross-lingual Language Model
|
||||
Pretraining <https://arxiv.org/abs/1901.07291>`__ by Guillaume Lample and Alexis Conneau.
|
||||
33. :doc:`XLM-ProphetNet <model_doc/xlmprophetnet>` (from Microsoft Research) released with the paper `ProphetNet:
|
||||
34. :doc:`XLM-ProphetNet <model_doc/xlmprophetnet>` (from Microsoft Research) released with the paper `ProphetNet:
|
||||
Predicting Future N-gram for Sequence-to-Sequence Pre-training <https://arxiv.org/abs/2001.04063>`__ by Yu Yan,
|
||||
Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang and Ming Zhou.
|
||||
34. :doc:`XLM-RoBERTa <model_doc/xlmroberta>` (from Facebook AI), released together with the paper `Unsupervised
|
||||
35. :doc:`XLM-RoBERTa <model_doc/xlmroberta>` (from Facebook AI), released together with the paper `Unsupervised
|
||||
Cross-lingual Representation Learning at Scale <https://arxiv.org/abs/1911.02116>`__ by Alexis Conneau*, Kartikay
|
||||
Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke
|
||||
Zettlemoyer and Veselin Stoyanov.
|
||||
35. :doc:`XLNet <model_doc/xlnet>` (from Google/CMU) released with the paper `XLNet: Generalized Autoregressive
|
||||
36. :doc:`XLNet <model_doc/xlnet>` (from Google/CMU) released with the paper `XLNet: Generalized Autoregressive
|
||||
Pretraining for Language Understanding <https://arxiv.org/abs/1906.08237>`__ by Zhilin Yang*, Zihang Dai*, Yiming
|
||||
Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.
|
||||
|
||||
@@ -382,6 +385,7 @@ TensorFlow and/or Flax.
|
||||
model_doc/roberta
|
||||
model_doc/squeezebert
|
||||
model_doc/t5
|
||||
model_doc/tapas
|
||||
model_doc/transformerxl
|
||||
model_doc/xlm
|
||||
model_doc/xlmprophetnet
|
||||
|
||||
@@ -0,0 +1,378 @@
|
||||
TAPAS
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The TAPAS model was proposed in `TAPAS: Weakly Supervised Table Parsing via Pre-training
|
||||
<https://arxiv.org/abs/2004.02349>`__ by Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno and
|
||||
Julian Martin Eisenschlos. It's a BERT-based model specifically designed (and pre-trained) for answering questions
|
||||
about tabular data. Compared to BERT, TAPAS uses relative position embeddings and has 7 token types that encode tabular
|
||||
structure. TAPAS is pre-trained on the masked language modeling (MLM) objective on a large dataset comprising millions
|
||||
of tables from English Wikipedia and corresponding texts. For question answering, TAPAS has 2 heads on top: a cell
|
||||
selection head and an aggregation head, for (optionally) performing aggregations (such as counting or summing) among
|
||||
selected cells. TAPAS has been fine-tuned on several datasets: SQA (Sequential Question Answering by Microsoft), WTQ
|
||||
(Wiki Table Questions by Stanford University) and WikiSQL (by Salesforce). It achieves state-of-the-art on both SQA and
|
||||
WTQ, while having comparable performance to SOTA on WikiSQL, with a much simpler architecture.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the
|
||||
collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations
|
||||
instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition,
|
||||
the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we
|
||||
present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak
|
||||
supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation
|
||||
operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective
|
||||
joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with
|
||||
three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by
|
||||
improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL
|
||||
and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our
|
||||
setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.*
|
||||
|
||||
In addition, the authors have further pre-trained TAPAS to recognize table entailment, by creating a balanced dataset
|
||||
of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning.
|
||||
The authors of TAPAS call this further pre-training intermediate pre-training (since TAPAS is first pre-trained on MLM,
|
||||
and then on another dataset). They found that intermediate pre-training further improves performance on SQA, achieving
|
||||
a new state-of-the-art as well as state-of-the-art on TabFact, a large-scale dataset with 16k Wikipedia tables for
|
||||
table entailment (a binary classification task). For more details, see their follow-up paper: `Understanding tables with
|
||||
intermediate pre-training <https://arxiv.org/abs/2010.00571>`__ by Julian Martin Eisenschlos, Syrine Krichene and
|
||||
Thomas Müller.
|
||||
|
||||
The original code can be found `here <https://github.com/google-research/tapas>`__.
|
||||
|
||||
Tips:
|
||||
|
||||
- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell
|
||||
of the table). According to the authors, this usually results in a slightly better performance, and allows you to
|
||||
encode longer sequences without running out of embeddings. This is reflected in the ``reset_position_index_per_cell``
|
||||
parameter of :class:`~transformers.TapasConfig`, which is set to ``True`` by default.
|
||||
There are both pre-trained models in the `model hub <https://huggingface.co/models?search=tapas>`_ with absolute and relative
|
||||
position embeddings. Note that it's usually advised to pad the inputs on the right rather than the left.
|
||||
- TAPAS is based on BERT, so ``TAPAS-base`` for example corresponds to a ``BERT-base`` architecture. Of course, TAPAS-large
|
||||
will result in the best performance (the results reported in the paper are from TAPAS-large). Metrics of the various
|
||||
sized models are shown on the `original Github repository <https://github.com/google-research/tapas>`_.
|
||||
- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a
|
||||
conversational set-up. This means that you can ask follow-up questions such as "what is his age?" related to the
|
||||
previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that
|
||||
case, you have to feed every training example one by one to the model, such that the `prev_label_ids` token type ids
|
||||
can be overwritten by the predicted `label_ids` of the model to the previous question. See "Usage" section for more info.
|
||||
- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore
|
||||
efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained
|
||||
with a causal language modeling (CLM) objective are better in that regard.
|
||||
|
||||
|
||||
Usage: fine-tuning
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Here we explain how you can fine-tune :class:`~transformers.TapasForQuestionAnswering` on your own dataset.
|
||||
|
||||
===========================================================================
|
||||
STEP 1: Choose one of the 3 ways in which you can use TAPAS - or experiment
|
||||
===========================================================================
|
||||
|
||||
Basically, there are 3 different ways in which one can fine-tune :class:`~transformers.TapasForQuestionAnswering`, corresponding to
|
||||
the different datasets on which Tapas was fine-tuned:
|
||||
|
||||
1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you
|
||||
first ask "what's the name of the first actor?" then you can ask a follow-up question such as "how old is he?". Here, questions
|
||||
do not involve any aggregation (all questions are cell selection questions).
|
||||
2. WTQ/WikiSQL: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related
|
||||
to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values.
|
||||
You can then for example ask "what's the total number of goals Cristiano Ronaldo made in his career?". This case is also called **weak
|
||||
supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer
|
||||
to the question as supervision.
|
||||
3. WikiSQL-supervised: this dataset is actually the same dataset as WikiSQL, but here the model is given the ground truth aggregation
|
||||
operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier.
|
||||
|
||||
To summarize:
|
||||
|
||||
+------------------------------------+----------------------+-------------------------------------------------------------------------------------------------------------------+
|
||||
| **Task** | **Example datasets** | **Description** |
|
||||
+------------------------------------+----------------------+-------------------------------------------------------------------------------------------------------------------+
|
||||
| Conversational | SQA | Conversational, only cell selection questions |
|
||||
+------------------------------------+----------------------+-------------------------------------------------------------------------------------------------------------------+
|
||||
| Weak supervision for aggregation | WTQ, WikiSQL | Questions might involve aggregation, and the model must learn this given only the answer as supervision |
|
||||
+------------------------------------+----------------------+-------------------------------------------------------------------------------------------------------------------+
|
||||
| Strong supervision for aggregation | WikiSQL-supervised | Questions might involve aggregation, and the model must learn this given the gold aggregation operator |
|
||||
+------------------------------------+----------------------+-------------------------------------------------------------------------------------------------------------------+
|
||||
|
||||
Initializing a model with a pre-trained base and randomly initialized classification heads from the model hub is as easy as:
|
||||
|
||||
.. code-block::
|
||||
|
||||
>>> from transformers import TapasForQuestionAnswering
|
||||
|
||||
>>> # for example, the base sized model
|
||||
>>> model = TapasForQuestionAnswering.from_pretrained('google/tapas-base-uncased')
|
||||
|
||||
|
||||
Of course, you don't necessarily have to follow one these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters
|
||||
you want when initializing :class:`~transformers.TapasConfig`, and then create a :class:`~transformers.TapasForQuestionAnswering` based on that
|
||||
configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it
|
||||
this way. Here's an example:
|
||||
|
||||
.. code-block::
|
||||
|
||||
>>> from transformers import TapasConfig, TapasForQuestionAnswering
|
||||
|
||||
>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)
|
||||
>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True, select_one_column=False)
|
||||
>>> # initializing the pre-trained base sized model with our custom classification heads
|
||||
>>> model = TapasForQuestionAnswering.from_pretrained('google/tapas-base-uncased', config=config)
|
||||
|
||||
What you can also do is start from an already fine-tuned checkpoint. A note here is that the already fine-tuned checkpoint on WTQ has some issues
|
||||
due to the L2-loss which is somewhat brittle. See `here <https://github.com/google-research/tapas/issues/91#issuecomment-735719340>`__ for more info.
|
||||
|
||||
For a list of all pre-trained and fine-tuned TAPAS checkpoints available in the HuggingFace model hub, see `here <https://huggingface.co/models?search=tapas>`__.
|
||||
|
||||
===========================================
|
||||
STEP 2: Prepare your data in the SQA format
|
||||
===========================================
|
||||
|
||||
Second, no matter what you picked above, you should prepare your dataset in the `SQA format <https://www.microsoft.com/en-us/download/details.aspx?id=54253>`__.
|
||||
This format is a TSV/CSV file with the following columns:
|
||||
|
||||
- ``id``: optional, id of the table-question pair, for bookkeeping purposes.
|
||||
- ``annotator``: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.
|
||||
- ``position``: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA).
|
||||
You don't need this column in case you're going for WTQ/WikiSQL/WikiSQL-supervised.
|
||||
- ``question``: string
|
||||
- ``table_file``: string, name of a csv file containing the tabular data
|
||||
- ``answer_coordinates``: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)
|
||||
- ``answer_text``: list of one or more strings (each string being a cell value that is part of the answer)
|
||||
- ``aggregation_label``: index of the aggregation operator. Only required in case of strong supervision for aggregation (the WikiSQL-supervised case)
|
||||
- ``float_answer``: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)
|
||||
|
||||
The tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion
|
||||
scripts with some automated logic to convert the other datasets (WTQ and WikiSQL) into the SQA format. The author explains this `here <https://github.com/google-research/tapas/issues/50#issuecomment-705465960>`__.
|
||||
Interestingly, these conversion scripts are not perfect (the ``answer_coordinates`` and ``float_answer`` fields are populated based on the ``answer_text``),
|
||||
meaning that WTQ and WikiSQL results could actually be improved.
|
||||
|
||||
|
||||
==========================================================================================
|
||||
STEP 3: Convert your data into PyTorch tensors using :class:`~transformers.TapasTokenizer`
|
||||
==========================================================================================
|
||||
|
||||
Third, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then
|
||||
use :class:`~transformers.TapasTokenizer` to convert table-question pairs into :obj:`input_ids`, :obj:`attention_mask`, :obj:`token_type_ids`
|
||||
and so on. Again, based on which of the three cases you picked above, :class:`~transformers.TapasForQuestionAnswering` requires different inputs
|
||||
to be fine-tuned:
|
||||
|
||||
+------------------------------------+----------------------------------------------------------------------------------------------+
|
||||
| **Task** | **Required inputs** |
|
||||
+------------------------------------+----------------------------------------------------------------------------------------------+
|
||||
| Conversational | ``input_ids``, ``attention_mask``, ``token_type_ids``, ``label_ids`` |
|
||||
+------------------------------------+----------------------------------------------------------------------------------------------+
|
||||
| Weak supervision for aggregation | ``input_ids``, ``attention_mask``, ``token_type_ids``, ``label_ids``, ``numeric_values``, |
|
||||
| | ``numeric_values_scale``, ``float_answer`` |
|
||||
+------------------------------------+----------------------------------------------------------------------------------------------+
|
||||
| Strong supervision for aggregation | ``input ids``, ``attention mask``, ``token type ids``, ``label ids``, ``aggregation_labels`` |
|
||||
+------------------------------------+----------------------------------------------------------------------------------------------+
|
||||
|
||||
:class:`~transformers.TapasTokenizer` creates the ``label_ids``, ``numeric_values`` and ``numeric_values_scale`` based on the
|
||||
``answer_coordinates`` and ``answer_text`` columns of the TSV file. The ``float_answer`` and ``aggregation_labels`` are already in the TSV file of step 2.
|
||||
Here's an example:
|
||||
|
||||
.. code-block::
|
||||
|
||||
>>> from transformers import TapasTokenizer
|
||||
>>> import pandas as pd
|
||||
|
||||
>>> model_name = 'google/tapas-base-uncased'
|
||||
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)
|
||||
|
||||
>>> data = {'Actors': ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], 'Number of movies': ["87", "53", "69"]}
|
||||
>>> queries = ["What is the name of the first actor?", "How many movies has George Clooney played in?", "What is the total number of movies?"]
|
||||
>>> answer_coordinates = [[(0, 0)], [(1, 0)], [(0, 2), (1, 2), (2, 2)]]
|
||||
>>> answer_text = [["Brad Pitt"], ["69"], ["209"]]
|
||||
>>> table = pd.Dataframe(data)
|
||||
>>> inputs = tokenizer(table=table, queries=queries, answer_coordinates=answer_coordinates, answer_text=answer_text, padding='max_length', return_tensors='pt')
|
||||
>>> inputs
|
||||
{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),
|
||||
'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), label_ids: tensor([[ ... ]])}
|
||||
|
||||
Note that :class:`~transformers.TapasTokenizer` expects the data of the table to be text-only. You can use ``.astype(str)`` on a dataframe to turn it into
|
||||
text-only data. Of course, this only shows how to encode a single training example. It is advised to create a PyTorch dataset and a corresponding dataloader:
|
||||
|
||||
.. code-block::
|
||||
|
||||
>>> import torch
|
||||
>>> import pandas as pd
|
||||
|
||||
>>> tsv_path = "your_path_to_the_tsv_file"
|
||||
>>> table_csv_path = "your_path_to_a_directory_containing_all_csv_files"
|
||||
|
||||
>>> class TableDataset(torch.utils.data.Dataset):
|
||||
... def __init__(self, data, tokenizer):
|
||||
... self.data = data
|
||||
... self.tokenizer = tokenizer
|
||||
...
|
||||
... def __getitem__(self, idx):
|
||||
... item = data.iloc[idx]
|
||||
... table = pd.read_csv(table_csv_path + item.table_file).astype(str)
|
||||
... encoding = self.tokenizer(table=table,
|
||||
... queries=item.question,
|
||||
... answer_coordinates=item.answer_coordinates,
|
||||
... answer_text=item.answer_text,
|
||||
... padding="max_length",
|
||||
... return_tensors="pt"
|
||||
... )
|
||||
... # we add the float_answer which is also required (weak supervision for aggregation)
|
||||
... encoding["float_answer"] = torch.tensor(item.float_answer)
|
||||
... return encoding
|
||||
...
|
||||
... def __len__(self):
|
||||
... return len(self.data)
|
||||
|
||||
>>> data = pd.read_csv(tsv_path, sep='\t')
|
||||
>>> train_dataset = TableDataset(data, tokenizer)
|
||||
>>> train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=32)
|
||||
|
||||
Note that here, we encode each table-question pair independently. This is fine as long as your dataset is **not conversational**. In case your
|
||||
dataset involves conversational questions (such as in SQA), then you should first group together the ``queries``, ``answer_coordinates`` and
|
||||
``answer_text`` per table (in the order of their ``position`` index) and batch encode each table with its questions. This will make sure that
|
||||
the ``prev_label_ids`` token types (see docs of :class:`~transformers.TapasTokenizer`) are set correctly.
|
||||
|
||||
===================================================
|
||||
STEP 4: Train (fine-tune) TapasForQuestionAnswering
|
||||
===================================================
|
||||
|
||||
You can then fine-tune :class:`~transformers.TapasForQuestionAnswering` using native PyTorch as follows:
|
||||
|
||||
.. code-block::
|
||||
|
||||
>>> from transformers import TapasForQuestionAnswering
|
||||
|
||||
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base-uncased")
|
||||
|
||||
>>> for epoch in range(2): # loop over the dataset multiple times
|
||||
... for idx, batch in enumerate(train_dataloader):
|
||||
... # get the inputs;
|
||||
... input_ids, attention_mask, token_type_ids, label_ids, numeric_values, numeric_values_scale, float_answer = batch
|
||||
|
||||
... # zero the parameter gradients
|
||||
... optimizer.zero_grad()
|
||||
|
||||
... # forward + backward + optimize
|
||||
... outputs = model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids,
|
||||
... label_ids=label_ids, numeric_values=numeric_values, numeric_values_scale=numeric_values_scale,
|
||||
... float_answer=float_answer)
|
||||
... loss = outputs.loss
|
||||
... loss.backward()
|
||||
... optimizer.step()
|
||||
|
||||
Usage: inference
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Here we explain how you can use :class:`~transformers.TapasForQuestionAnswering` for inference (i.e. making predictions on new data).
|
||||
For inference, only ``input_ids``, ``attention_mask`` and ``token_type_ids`` (which you can obtain using
|
||||
:class:`~transformers.TapasTokenizer`) have to provided to the model to obtain the logits. Next, you can use the handy
|
||||
``convert_logits_to_predictions`` method of :class:`~transformers.TapasTokenizer` to convert these into predicted coordinates
|
||||
and optional aggregation indices.
|
||||
|
||||
However, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference
|
||||
can be done in parallel on all table-question pairs of a batch. Here's an example of that:
|
||||
|
||||
.. code-block::
|
||||
|
||||
>>> from transformers import TapasTokenizer, TapasForQuestionAnswering
|
||||
>>> import pandas as pd
|
||||
|
||||
>>> model_name = 'google/tapas-base-uncased-finetuned-wtq'
|
||||
>>> model = TapasForQuestionAnswering.from_pretrained(model_name)
|
||||
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)
|
||||
|
||||
>>> data = {'Actors': ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], 'Number of movies': ["87", "53", "69"]}
|
||||
>>> queries = ["What is the name of the first actor?", "How many movies has George Clooney played in?", "What is the total number of movies?"]
|
||||
>>> table = pd.Dataframe(data)
|
||||
>>> inputs = tokenizer(table=table, queries=queries, padding='max_length', return_tensors="pt")
|
||||
>>> outputs = model(**inputs)
|
||||
>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(
|
||||
... inputs,
|
||||
... output.logits,
|
||||
... outputs.logits_aggregation
|
||||
...)
|
||||
|
||||
>>> # let's print out the results:
|
||||
>>> id2aggregation = {0: "NONE", 1: "SUM", 2: "AVERAGE", 3:"COUNT"}
|
||||
>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]
|
||||
|
||||
>>> answers = []
|
||||
>>> for coordinates in predicted_answer_coordinates:
|
||||
... if len(coordinates) == 1:
|
||||
... # only a single cell:
|
||||
... answers.append(df.iat[coordinates[0]])
|
||||
... else:
|
||||
... # multiple cells
|
||||
... cell_values = []
|
||||
... for coordinate in coordinates:
|
||||
... cell_values.append(df.iat[coordinate])
|
||||
... answers.append(", ".join(cell_values))
|
||||
|
||||
>>> display(df)
|
||||
>>> print("")
|
||||
>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):
|
||||
... print(query)
|
||||
... if predicted_agg == "NONE":
|
||||
... print("Predicted answer: " + answer)
|
||||
... else:
|
||||
... print("Predicted answer: " + predicted_agg + " > " + answer)
|
||||
When was Brad Pitt born?
|
||||
Predicted answer: 18 december 1963
|
||||
Which actor appeared in the least number of movies?
|
||||
Predicted answer: Leonardo Di Caprio
|
||||
What is the average number of movies?
|
||||
Predicted answer: AVERAGE > 87, 53, 69
|
||||
|
||||
In case of a conversational set-up, then each table-question pair must be provided **sequentially** to the model, such that
|
||||
the ``prev_label_ids`` token types can be overwritten by the predicted ``label_ids`` of the previous table-question pair.
|
||||
|
||||
|
||||
Tapas specific outputs
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.modeling_tapas.TableQuestionAnsweringOutput
|
||||
:members:
|
||||
|
||||
|
||||
TapasConfig
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TapasConfig
|
||||
:members:
|
||||
|
||||
|
||||
TapasTokenizer
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TapasTokenizer
|
||||
:members: __call__, convert_logits_to_predictions, save_vocabulary
|
||||
|
||||
|
||||
TapasModel
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TapasModel
|
||||
:members:
|
||||
|
||||
|
||||
TapasForMaskedLM
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TapasForMaskedLM
|
||||
:members:
|
||||
|
||||
|
||||
TapasForSequenceClassification
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TapasForSequenceClassification
|
||||
:members: forward
|
||||
|
||||
|
||||
TapasForQuestionAnswering
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TapasForQuestionAnswering
|
||||
:members:
|
||||
@@ -164,6 +164,7 @@ from .models.retribert import RETRIBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, RetriBert
|
||||
from .models.roberta import ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, RobertaConfig, RobertaTokenizer
|
||||
from .models.squeezebert import SQUEEZEBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, SqueezeBertConfig, SqueezeBertTokenizer
|
||||
from .models.t5 import T5_PRETRAINED_CONFIG_ARCHIVE_MAP, T5Config
|
||||
from .models.tapas import TAPAS_PRETRAINED_CONFIG_ARCHIVE_MAP, TapasConfig, TapasTokenizer
|
||||
from .models.transfo_xl import (
|
||||
TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
TransfoXLConfig,
|
||||
@@ -605,6 +606,13 @@ if is_torch_available():
|
||||
T5PreTrainedModel,
|
||||
load_tf_weights_in_t5,
|
||||
)
|
||||
from .models.tapas import(
|
||||
TAPAS_PRETRAINED_MODEL_ARCHIVE_LIST,
|
||||
TapasModel,
|
||||
TapasForMaskedLM,
|
||||
TapasForQuestionAnswering,
|
||||
TapasForSequenceClassification,
|
||||
)
|
||||
from .models.transfo_xl import (
|
||||
TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_LIST,
|
||||
AdaptiveEmbedding,
|
||||
|
||||
@@ -51,6 +51,7 @@ from ..retribert.configuration_retribert import RETRIBERT_PRETRAINED_CONFIG_ARCH
|
||||
from ..roberta.configuration_roberta import ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, RobertaConfig
|
||||
from ..squeezebert.configuration_squeezebert import SQUEEZEBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, SqueezeBertConfig
|
||||
from ..t5.configuration_t5 import T5_PRETRAINED_CONFIG_ARCHIVE_MAP, T5Config
|
||||
from ..tapas.configuration_tapas import TAPAS_PRETRAINED_CONFIG_ARCHIVE_MAP, TapasConfig
|
||||
from ..transfo_xl.configuration_transfo_xl import TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP, TransfoXLConfig
|
||||
from ..xlm.configuration_xlm import XLM_PRETRAINED_CONFIG_ARCHIVE_MAP, XLMConfig
|
||||
from ..xlm_prophetnet.configuration_xlm_prophetnet import (
|
||||
@@ -95,6 +96,7 @@ ALL_PRETRAINED_CONFIG_ARCHIVE_MAP = dict(
|
||||
XLM_PROPHETNET_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
PROPHETNET_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
MPNET_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
TAPAS_PRETRAINED_CONFIG_ARCHIVE_MAP,
|
||||
]
|
||||
for key, value, in pretrained_map.items()
|
||||
)
|
||||
@@ -141,6 +143,7 @@ CONFIG_MAPPING = OrderedDict(
|
||||
("dpr", DPRConfig),
|
||||
("layoutlm", LayoutLMConfig),
|
||||
("rag", RagConfig),
|
||||
("tapas", TapasConfig),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -185,6 +188,7 @@ MODEL_NAMES_MAPPING = OrderedDict(
|
||||
("prophetnet", "ProphetNet"),
|
||||
("mt5", "mT5"),
|
||||
("mpnet", "MPNet"),
|
||||
("tapas", "TAPAS"),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@@ -165,6 +165,12 @@ from ..squeezebert.modeling_squeezebert import (
|
||||
SqueezeBertModel,
|
||||
)
|
||||
from ..t5.modeling_t5 import T5ForConditionalGeneration, T5Model
|
||||
from ..tapas.modeling_tapas import (
|
||||
TapasModel,
|
||||
TapasForMaskedLM,
|
||||
TapasForQuestionAnswering,
|
||||
TapasForSequenceClassification,
|
||||
)
|
||||
from ..transfo_xl.modeling_transfo_xl import TransfoXLForSequenceClassification, TransfoXLLMHeadModel, TransfoXLModel
|
||||
from ..xlm.modeling_xlm import (
|
||||
XLMForMultipleChoice,
|
||||
@@ -230,6 +236,7 @@ from .configuration_auto import (
|
||||
RobertaConfig,
|
||||
SqueezeBertConfig,
|
||||
T5Config,
|
||||
TapasConfig,
|
||||
TransfoXLConfig,
|
||||
XLMConfig,
|
||||
XLMProphetNetConfig,
|
||||
@@ -277,6 +284,7 @@ MODEL_MAPPING = OrderedDict(
|
||||
(XLMProphetNetConfig, XLMProphetNetModel),
|
||||
(ProphetNetConfig, ProphetNetModel),
|
||||
(MPNetConfig, MPNetModel),
|
||||
(TapasConfig, TapasModel),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -308,6 +316,7 @@ MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
|
||||
(LxmertConfig, LxmertForPreTraining),
|
||||
(FunnelConfig, FunnelForPreTraining),
|
||||
(MPNetConfig, MPNetForMaskedLM),
|
||||
(TapasConfig, TapasForMaskedLM),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -340,6 +349,7 @@ MODEL_WITH_LM_HEAD_MAPPING = OrderedDict(
|
||||
(ReformerConfig, ReformerModelWithLMHead),
|
||||
(FunnelConfig, FunnelForMaskedLM),
|
||||
(MPNetConfig, MPNetForMaskedLM),
|
||||
(TapasConfig, TapasForMaskedLM),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -386,6 +396,7 @@ MODEL_FOR_MASKED_LM_MAPPING = OrderedDict(
|
||||
(ReformerConfig, ReformerForMaskedLM),
|
||||
(FunnelConfig, FunnelForMaskedLM),
|
||||
(MPNetConfig, MPNetForMaskedLM),
|
||||
(TapasConfig, TapasForMaskedLM),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -431,6 +442,7 @@ MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING = OrderedDict(
|
||||
(CTRLConfig, CTRLForSequenceClassification),
|
||||
(TransfoXLConfig, TransfoXLForSequenceClassification),
|
||||
(MPNetConfig, MPNetForSequenceClassification),
|
||||
(TapasConfig, TapasForSequenceClassification),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -455,6 +467,7 @@ MODEL_FOR_QUESTION_ANSWERING_MAPPING = OrderedDict(
|
||||
(FunnelConfig, FunnelForQuestionAnswering),
|
||||
(LxmertConfig, LxmertForQuestionAnswering),
|
||||
(MPNetConfig, MPNetForQuestionAnswering),
|
||||
(TapasConfig, TapasForQuestionAnswering),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@@ -47,6 +47,7 @@ from ..rag.tokenization_rag import RagTokenizer
|
||||
from ..retribert.tokenization_retribert import RetriBertTokenizer
|
||||
from ..roberta.tokenization_roberta import RobertaTokenizer
|
||||
from ..squeezebert.tokenization_squeezebert import SqueezeBertTokenizer
|
||||
from ..tapas.tokenization_tapas import TapasTokenizer
|
||||
from ..transfo_xl.tokenization_transfo_xl import TransfoXLTokenizer
|
||||
from ..xlm.tokenization_xlm import XLMTokenizer
|
||||
from .configuration_auto import (
|
||||
@@ -84,6 +85,7 @@ from .configuration_auto import (
|
||||
RobertaConfig,
|
||||
SqueezeBertConfig,
|
||||
T5Config,
|
||||
TapasConfig,
|
||||
TransfoXLConfig,
|
||||
XLMConfig,
|
||||
XLMProphetNetConfig,
|
||||
@@ -223,6 +225,7 @@ TOKENIZER_MAPPING = OrderedDict(
|
||||
(XLMProphetNetConfig, (XLMProphetNetTokenizer, None)),
|
||||
(ProphetNetConfig, (ProphetNetTokenizer, None)),
|
||||
(MPNetConfig, (MPNetTokenizer, MPNetTokenizerFast)),
|
||||
(TapasConfig, (TapasTokenizer, None)),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2020 Google Research and The HuggingFace Inc. team.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
""" TAPAS configuration. Adds additional hyperparameters to the configuration of BERT."""
|
||||
|
||||
|
||||
from .configuration_utils import PretrainedConfig
|
||||
|
||||
|
||||
TAPAS_PRETRAINED_CONFIG_ARCHIVE_MAP = {"nielsr/tapas-base-finetuned-sqa": "https://huggingface.co/nielsr/tapas-base-finetuned-sqa/resolve/main/config.json",
|
||||
"nielsr/tapas-base-finetuned-wtq": "https://huggingface.co/nielsr/tapas-base-finetuned-wtq/resolve/main/config.json",
|
||||
"nielsr/tapas-base-finetuned-wikisql-supervised": "https://huggingface.co/nielsr/tapas-base-finetuned-wikisql-supervised/resolve/main/config.json",
|
||||
"nielsr/tapas-base-finetuned-tabfact": "https://huggingface.co/nielsr/tapas-base-finetuned-tabfact/resolve/main/config.json"}
|
||||
|
||||
|
||||
class TapasConfig(PretrainedConfig):
|
||||
r"""
|
||||
This is the configuration class to store the configuration of a :class:`~transformers.TapasModel`. It is used to
|
||||
instantiate a TAPAS model according to the specified arguments, defining the model architecture. Instantiating a
|
||||
configuration with the defaults will yield a similar configuration to that of the TAPAS `tapas-base-finetuned-sqa`
|
||||
architecture. Configuration objects inherit from :class:`~transformers.PreTrainedConfig` and can be used to control
|
||||
the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Hyperparameters additional to BERT are taken from run_task_main.py and hparam_utils.py of the original
|
||||
implementation. Original implementation available at https://github.com/google-research/tapas/tree/master.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 30522):
|
||||
Vocabulary size of the TAPAS model. Defines the number of different tokens that can be represented by the
|
||||
:obj:`inputs_ids` passed when calling :class:`~transformers.TapasModel`.
|
||||
hidden_size (:obj:`int`, `optional`, defaults to 768):
|
||||
Dimensionality of the encoder layers and the pooler layer.
|
||||
num_hidden_layers (:obj:`int`, `optional`, defaults to 12):
|
||||
Number of hidden layers in the Transformer encoder.
|
||||
num_attention_heads (:obj:`int`, `optional`, defaults to 12):
|
||||
Number of attention heads for each attention layer in the Transformer encoder.
|
||||
intermediate_size (:obj:`int`, `optional`, defaults to 3072):
|
||||
Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
|
||||
hidden_act (:obj:`str` or :obj:`Callable`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
hidden_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_probs_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout ratio for the attention probabilities.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 1024):
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
type_vocab_sizes (:obj:`List[int]`, `optional`, defaults to [3, 256, 256, 2, 256, 256, 10]):
|
||||
The vocabulary sizes of the :obj:`token_type_ids` passed when calling :class:`~transformers.TapasModel`.
|
||||
initializer_range (:obj:`float`, `optional`, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
layer_norm_eps (:obj:`float`, `optional`, defaults to 1e-12):
|
||||
The epsilon used by the layer normalization layers.
|
||||
gradient_checkpointing (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether to use gradient checkpointing to save memory at the expense of a slower backward pass.
|
||||
positive_label_weight (:obj:`float`, `optional`, defaults to 10.0):
|
||||
Weight for positive labels.
|
||||
num_aggregation_labels (:obj:`int`, `optional`, defaults to 0):
|
||||
The number of aggregation operators to predict.
|
||||
aggregation_loss_weight (:obj:`float`, `optional`, defaults to 1.0):
|
||||
Importance weight for the aggregation loss.
|
||||
use_answer_as_supervision (:obj:`bool`, `optional`):
|
||||
Whether to use the answer as the only supervision for aggregation examples.
|
||||
answer_loss_importance (:obj:`float`, `optional`, defaults to 1.0):
|
||||
Importance weight for the regression loss.
|
||||
use_normalized_answer_loss (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether to normalize the answer loss by the maximum of the predicted and expected value.
|
||||
huber_loss_delta: (:obj:`float`, `optional`):
|
||||
Delta parameter used to calculate the regression loss.
|
||||
temperature: (:obj:`float`, `optional`, defaults to 1.0):
|
||||
Value used to control (OR change) the skewness of cell logits probabilities.
|
||||
aggregation_temperature: (:obj:`float`, `optional`, defaults to 1.0):
|
||||
Scales aggregation logits to control the skewness of probabilities.
|
||||
use_gumbel_for_cells: (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether to apply Gumbel-Softmax to cell selection.
|
||||
use_gumbel_for_aggregation: (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether to apply Gumbel-Softmax to aggregation selection.
|
||||
average_approximation_function: (:obj:`string`, `optional`, defaults to :obj:`"ratio"`):
|
||||
Method to calculate the expected average of cells in the weak supervision case. One of :obj:`"ratio"`,
|
||||
:obj:`"first_order"` or :obj:`"second_order"`.
|
||||
cell_selection_preference: (:obj:`float`, `optional`):
|
||||
Preference for cell selection in ambiguous cases. Only applicable in case of weak supervision for
|
||||
aggregation (WTQ, WikiSQL). If the total mass of the aggregation probabilities (excluding the "NONE"
|
||||
operator) is higher than this hyperparameter, then aggregation is predicted for an example.
|
||||
answer_loss_cutoff: (:obj:`float`, `optional`):
|
||||
Ignore examples with answer loss larger than cutoff.
|
||||
max_num_rows: (:obj:`int`, `optional`, defaults to 64):
|
||||
Maximum number of rows.
|
||||
max_num_columns: (:obj:`int`, `optional`, defaults to 32):
|
||||
Maximum number of columns.
|
||||
average_logits_per_cell: (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether to average logits per cell.
|
||||
select_one_column: (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
Whether to constrain the model to only select cells from a single column.
|
||||
allow_empty_column_selection: (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether to allow not to select any column.
|
||||
init_cell_selection_weights_to_zero: (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether to initialize cell selection weights to 0 so that the initial probabilities are 50%.
|
||||
reset_position_index_per_cell: (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
Whether to restart position indexes at every cell (i.e. use relative position embeddings).
|
||||
disable_per_token_loss: (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether to disable any (strong or weak) supervision on cells.
|
||||
|
||||
Example::
|
||||
|
||||
>>> from transformers import TapasModel, TapasConfig
|
||||
>>> # Initializing a Tapas configuration
|
||||
>>> configuration = TapasConfig()
|
||||
>>> # Initializing a model from the configuration
|
||||
>>> model = TapasModel(configuration)
|
||||
>>> # Accessing the model configuration
|
||||
>>> configuration = model.config
|
||||
"""
|
||||
|
||||
model_type = "tapas"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vocab_size=30522,
|
||||
hidden_size=768,
|
||||
num_hidden_layers=12,
|
||||
num_attention_heads=12,
|
||||
intermediate_size=3072,
|
||||
hidden_act="gelu",
|
||||
hidden_dropout_prob=0.1,
|
||||
attention_probs_dropout_prob=0.1,
|
||||
max_position_embeddings=1024,
|
||||
type_vocab_sizes=[3, 256, 256, 2, 256, 256, 10],
|
||||
initializer_range=0.02,
|
||||
layer_norm_eps=1e-12,
|
||||
pad_token_id=0,
|
||||
gradient_checkpointing=False,
|
||||
positive_label_weight=10.0,
|
||||
num_aggregation_labels=0,
|
||||
aggregation_loss_weight=1.0,
|
||||
use_answer_as_supervision=None,
|
||||
answer_loss_importance=1.0,
|
||||
use_normalized_answer_loss=False,
|
||||
huber_loss_delta=None,
|
||||
temperature=1.0,
|
||||
aggregation_temperature=1.0,
|
||||
use_gumbel_for_cells=False,
|
||||
use_gumbel_for_aggregation=False,
|
||||
average_approximation_function="ratio",
|
||||
cell_selection_preference=None,
|
||||
answer_loss_cutoff=None,
|
||||
max_num_rows=64,
|
||||
max_num_columns=32,
|
||||
average_logits_per_cell=False,
|
||||
select_one_column=True,
|
||||
allow_empty_column_selection=False,
|
||||
init_cell_selection_weights_to_zero=False,
|
||||
reset_position_index_per_cell=True,
|
||||
disable_per_token_loss=False,
|
||||
**kwargs
|
||||
):
|
||||
|
||||
super().__init__(pad_token_id=pad_token_id, **kwargs)
|
||||
|
||||
# BERT hyperparameters (with updated max_position_embeddings and type_vocab_sizes)
|
||||
self.vocab_size = vocab_size
|
||||
self.hidden_size = hidden_size
|
||||
self.num_hidden_layers = num_hidden_layers
|
||||
self.num_attention_heads = num_attention_heads
|
||||
self.hidden_act = hidden_act
|
||||
self.intermediate_size = intermediate_size
|
||||
self.hidden_dropout_prob = hidden_dropout_prob
|
||||
self.attention_probs_dropout_prob = attention_probs_dropout_prob
|
||||
self.max_position_embeddings = max_position_embeddings
|
||||
self.type_vocab_sizes = type_vocab_sizes
|
||||
self.initializer_range = initializer_range
|
||||
self.layer_norm_eps = layer_norm_eps
|
||||
self.gradient_checkpointing = gradient_checkpointing
|
||||
|
||||
# Fine-tuning task hyperparameters
|
||||
self.positive_label_weight = positive_label_weight
|
||||
self.num_aggregation_labels = num_aggregation_labels
|
||||
self.aggregation_loss_weight = aggregation_loss_weight
|
||||
self.use_answer_as_supervision = use_answer_as_supervision
|
||||
self.answer_loss_importance = answer_loss_importance
|
||||
self.use_normalized_answer_loss = use_normalized_answer_loss
|
||||
self.huber_loss_delta = huber_loss_delta
|
||||
self.temperature = temperature
|
||||
self.aggregation_temperature = aggregation_temperature
|
||||
self.use_gumbel_for_cells = use_gumbel_for_cells
|
||||
self.use_gumbel_for_aggregation = use_gumbel_for_aggregation
|
||||
self.average_approximation_function = average_approximation_function
|
||||
self.cell_selection_preference = cell_selection_preference
|
||||
self.answer_loss_cutoff = answer_loss_cutoff
|
||||
self.max_num_rows = max_num_rows
|
||||
self.max_num_columns = max_num_columns
|
||||
self.average_logits_per_cell = average_logits_per_cell
|
||||
self.select_one_column = select_one_column
|
||||
self.allow_empty_column_selection = allow_empty_column_selection
|
||||
self.init_cell_selection_weights_to_zero = init_cell_selection_weights_to_zero
|
||||
self.reset_position_index_per_cell = reset_position_index_per_cell
|
||||
self.disable_per_token_loss = disable_per_token_loss
|
||||
@@ -0,0 +1,120 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2018 The HuggingFace Inc. team.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
"""Convert TAPAS checkpoint."""
|
||||
|
||||
|
||||
import argparse
|
||||
|
||||
import torch
|
||||
|
||||
from transformers import (
|
||||
TapasConfig,
|
||||
TapasModel,
|
||||
TapasForQuestionAnswering,
|
||||
TapasForSequenceClassification,
|
||||
load_tf_weights_in_tapas,
|
||||
)
|
||||
from transformers.utils import logging
|
||||
|
||||
|
||||
logging.set_verbosity_info()
|
||||
|
||||
|
||||
def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, tapas_config_file, pytorch_dump_path):
|
||||
# Initialise PyTorch model. Defaults to TapasForQuestionAnswering with default SQA config.
|
||||
# Uncomment another config and/or model to change this. If you want to convert a checkpoint
|
||||
# that has absolute position embeddings, make sure to set reset_position_index_per_cell of
|
||||
# TapasConfig to False.
|
||||
|
||||
# WTQ config
|
||||
# config = TapasConfig(
|
||||
# # run_task_main.py hparams
|
||||
# num_aggregation_labels = 4,
|
||||
# use_answer_as_supervision = True,
|
||||
# # hparam_utils.py hparams
|
||||
# answer_loss_cutoff = 0.664694,
|
||||
# cell_selection_preference = 0.207951,
|
||||
# huber_loss_delta = 0.121194,
|
||||
# init_cell_selection_weights_to_zero = True,
|
||||
# select_one_column = True,
|
||||
# allow_empty_column_selection = False,
|
||||
# temperature = 0.0352513,
|
||||
# )
|
||||
|
||||
# WikiSQL config
|
||||
# config = TapasConfig(
|
||||
# # run_task_main.py hparams
|
||||
# num_aggregation_labels = 4,
|
||||
# use_answer_as_supervision = True,
|
||||
# # hparam_utils.py hparams
|
||||
# answer_loss_cutoff = 0.185567,
|
||||
# cell_selection_preference = 0.611754,
|
||||
# huber_loss_delta = 1265.74,
|
||||
# init_cell_selection_weights_to_zero = False,
|
||||
# select_one_column = False,
|
||||
# allow_empty_column_selection = False,
|
||||
# temperature = 0.107515,
|
||||
# )
|
||||
|
||||
# WikiSQL-supervised config
|
||||
# config = TapasConfig(
|
||||
# # run_task_main.py hparams
|
||||
# num_aggregation_labels = 4,
|
||||
# use_answer_as_supervision = False,
|
||||
# # hparam_utils.py hparams
|
||||
# answer_loss_cutoff = 36.4519,
|
||||
# cell_selection_preference = 0.903421,
|
||||
# huber_loss_delta = 222.088,
|
||||
# init_cell_selection_weights_to_zero = True,
|
||||
# select_one_column = True,
|
||||
# allow_empty_column_selection = True,
|
||||
# temperature = 0.763141,
|
||||
# )
|
||||
|
||||
# SQA config
|
||||
config = TapasConfig()
|
||||
|
||||
print("Building PyTorch model from configuration: {}".format(str(config)))
|
||||
model = TapasModel(config)
|
||||
#model = TapasForQuestionAnswering(config)
|
||||
# model = TapasForSequenceClassification(config)
|
||||
|
||||
# Load weights from tf checkpoint
|
||||
load_tf_weights_in_tapas(model, config, tf_checkpoint_path)
|
||||
|
||||
# Save pytorch-model
|
||||
print("Save PyTorch model to {}".format(pytorch_dump_path))
|
||||
torch.save(model.state_dict(), pytorch_dump_path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
# Required parameters
|
||||
parser.add_argument(
|
||||
"--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--tapas_config_file",
|
||||
default=None,
|
||||
type=str,
|
||||
required=True,
|
||||
help="The config json file corresponding to the pre-trained TAPAS model. \n"
|
||||
"This specifies the model architecture.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model."
|
||||
)
|
||||
args = parser.parse_args()
|
||||
convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.tapas_config_file, args.pytorch_dump_path)
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,863 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2020 Google Research and The HuggingFace Inc. team.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
import unittest
|
||||
|
||||
import copy
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from transformers import is_torch_available
|
||||
from transformers.file_utils import cached_property
|
||||
from transformers.testing_utils import require_torch, slow, torch_device
|
||||
|
||||
from .test_configuration_common import ConfigTester
|
||||
from .test_modeling_common import ModelTesterMixin, floats_tensor, ids_tensor, random_attention_mask
|
||||
|
||||
|
||||
if is_torch_available():
|
||||
import torch
|
||||
|
||||
from transformers import (
|
||||
TAPAS_PRETRAINED_MODEL_ARCHIVE_LIST,
|
||||
TapasConfig,
|
||||
#TapasForMaskedLM,
|
||||
TapasForQuestionAnswering,
|
||||
TapasForSequenceClassification,
|
||||
TapasModel,
|
||||
)
|
||||
|
||||
from transformers.modeling_tapas import (
|
||||
IndexMap,
|
||||
ProductIndexMap,
|
||||
gather,
|
||||
flatten,
|
||||
range_index_map,
|
||||
reduce_sum,
|
||||
reduce_mean,
|
||||
reduce_max,
|
||||
reduce_min,
|
||||
)
|
||||
|
||||
|
||||
class TapasModelTester:
|
||||
"""You can also import this e.g from .test_modeling_tapas import TapasModelTester """
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
parent,
|
||||
batch_size=13,
|
||||
seq_length=7,
|
||||
is_training=True,
|
||||
use_input_mask=True,
|
||||
use_token_type_ids=True,
|
||||
use_labels=True,
|
||||
vocab_size=99,
|
||||
hidden_size=32,
|
||||
num_hidden_layers=5,
|
||||
num_attention_heads=4,
|
||||
intermediate_size=37,
|
||||
hidden_act="gelu",
|
||||
hidden_dropout_prob=0.1,
|
||||
attention_probs_dropout_prob=0.1,
|
||||
initializer_range=0.02,
|
||||
max_position_embeddings=512,
|
||||
type_vocab_sizes=[3, 256, 256, 2, 256, 256, 10],
|
||||
type_sequence_label_size=2,
|
||||
positive_weight=10.0,
|
||||
num_aggregation_labels=4,
|
||||
num_labels=2,
|
||||
aggregation_loss_importance=0.8,
|
||||
use_answer_as_supervision=True,
|
||||
answer_loss_importance=0.001,
|
||||
use_normalized_answer_loss=False,
|
||||
huber_loss_delta=25.0,
|
||||
temperature=1.0,
|
||||
agg_temperature=1.0,
|
||||
use_gumbel_for_cells=False,
|
||||
use_gumbel_for_agg=False,
|
||||
average_approximation_function="ratio",
|
||||
cell_selection_preference=0.5,
|
||||
answer_loss_cutoff=100,
|
||||
max_num_rows=64,
|
||||
max_num_columns=32,
|
||||
average_logits_per_cell=True,
|
||||
select_one_column=True,
|
||||
allow_empty_column_selection=False,
|
||||
init_cell_selection_weights_to_zero=False,
|
||||
reset_position_index_per_cell=True,
|
||||
disable_per_token_loss=False,
|
||||
scope=None,
|
||||
):
|
||||
self.parent = parent
|
||||
self.batch_size = batch_size
|
||||
self.seq_length = seq_length
|
||||
self.is_training = is_training
|
||||
self.use_input_mask = use_input_mask
|
||||
self.use_token_type_ids = use_token_type_ids
|
||||
self.use_labels = use_labels
|
||||
self.vocab_size = vocab_size
|
||||
self.hidden_size = hidden_size
|
||||
self.num_hidden_layers = num_hidden_layers
|
||||
self.num_attention_heads = num_attention_heads
|
||||
self.intermediate_size = intermediate_size
|
||||
self.hidden_act = hidden_act
|
||||
self.hidden_dropout_prob = hidden_dropout_prob
|
||||
self.attention_probs_dropout_prob = attention_probs_dropout_prob
|
||||
self.initializer_range = initializer_range
|
||||
self.max_position_embeddings = max_position_embeddings
|
||||
self.type_vocab_sizes = type_vocab_sizes
|
||||
self.type_sequence_label_size = type_sequence_label_size
|
||||
self.positive_weight = positive_weight
|
||||
self.num_aggregation_labels = num_aggregation_labels
|
||||
self.num_labels = num_labels
|
||||
self.aggregation_loss_importance = aggregation_loss_importance
|
||||
self.use_answer_as_supervision = use_answer_as_supervision
|
||||
self.answer_loss_importance = answer_loss_importance
|
||||
self.use_normalized_answer_loss = use_normalized_answer_loss
|
||||
self.huber_loss_delta = huber_loss_delta
|
||||
self.temperature = temperature
|
||||
self.agg_temperature = agg_temperature
|
||||
self.use_gumbel_for_cells = use_gumbel_for_cells
|
||||
self.use_gumbel_for_agg = use_gumbel_for_agg
|
||||
self.average_approximation_function = average_approximation_function
|
||||
self.cell_selection_preference = cell_selection_preference
|
||||
self.answer_loss_cutoff = answer_loss_cutoff
|
||||
self.max_num_rows = max_num_rows
|
||||
self.max_num_columns = max_num_columns
|
||||
self.average_logits_per_cell = average_logits_per_cell
|
||||
self.select_one_column = select_one_column
|
||||
self.allow_empty_column_selection = allow_empty_column_selection
|
||||
self.init_cell_selection_weights_to_zero = init_cell_selection_weights_to_zero
|
||||
self.reset_position_index_per_cell = reset_position_index_per_cell
|
||||
self.disable_per_token_loss = disable_per_token_loss
|
||||
self.scope = scope
|
||||
|
||||
def prepare_config_and_inputs(self):
|
||||
input_ids = ids_tensor([self.batch_size, self.seq_length], self.vocab_size)
|
||||
|
||||
input_mask = None
|
||||
if self.use_input_mask:
|
||||
input_mask = random_attention_mask([self.batch_size, self.seq_length])
|
||||
|
||||
token_type_ids = []
|
||||
for type_vocab_size in self.type_vocab_sizes:
|
||||
token_type_ids.append(ids_tensor(shape=[self.batch_size, self.seq_length], vocab_size=type_vocab_size))
|
||||
token_type_ids = torch.stack(token_type_ids, dim=2)
|
||||
|
||||
sequence_labels = None
|
||||
token_labels = None
|
||||
label_ids = None
|
||||
answer = None
|
||||
numeric_values = None
|
||||
numeric_values_scale = None
|
||||
float_answer = None
|
||||
aggregation_labels = None
|
||||
if self.use_labels:
|
||||
sequence_labels = ids_tensor([self.batch_size], self.type_sequence_label_size)
|
||||
token_labels = ids_tensor([self.batch_size, self.seq_length], self.num_labels)
|
||||
label_ids = ids_tensor([self.batch_size, self.seq_length], vocab_size=2)
|
||||
numeric_values = floats_tensor([self.batch_size, self.seq_length])
|
||||
numeric_values_scale = floats_tensor([self.batch_size, self.seq_length])
|
||||
float_answer = floats_tensor([self.batch_size])
|
||||
aggregation_labels = ids_tensor([self.batch_size], self.num_aggregation_labels)
|
||||
|
||||
config = TapasConfig(
|
||||
vocab_size=self.vocab_size,
|
||||
hidden_size=self.hidden_size,
|
||||
num_hidden_layers=self.num_hidden_layers,
|
||||
num_attention_heads=self.num_attention_heads,
|
||||
intermediate_size=self.intermediate_size,
|
||||
hidden_act=self.hidden_act,
|
||||
hidden_dropout_prob=self.hidden_dropout_prob,
|
||||
attention_probs_dropout_prob=self.attention_probs_dropout_prob,
|
||||
max_position_embeddings=self.max_position_embeddings,
|
||||
type_vocab_sizes=self.type_vocab_sizes,
|
||||
initializer_range=self.initializer_range,
|
||||
positive_weight=self.positive_weight,
|
||||
num_aggregation_labels=self.num_aggregation_labels,
|
||||
num_labels=self.num_labels,
|
||||
aggregation_loss_importance=self.aggregation_loss_importance,
|
||||
use_answer_as_supervision=self.use_answer_as_supervision,
|
||||
answer_loss_importance=self.answer_loss_importance,
|
||||
use_normalized_answer_loss=self.use_normalized_answer_loss,
|
||||
huber_loss_delta=self.huber_loss_delta,
|
||||
temperature=self.temperature,
|
||||
agg_temperature=self.agg_temperature,
|
||||
use_gumbel_for_cells=self.use_gumbel_for_cells,
|
||||
use_gumbel_for_agg=self.use_gumbel_for_agg,
|
||||
average_approximation_function=self.average_approximation_function,
|
||||
cell_selection_preference=self.cell_selection_preference,
|
||||
answer_loss_cutoff=self.answer_loss_cutoff,
|
||||
max_num_rows=self.max_num_rows,
|
||||
max_num_columns=self.max_num_columns,
|
||||
average_logits_per_cell=self.average_logits_per_cell,
|
||||
select_one_column=self.select_one_column,
|
||||
allow_empty_column_selection=self.allow_empty_column_selection,
|
||||
init_cell_selection_weights_to_zero=self.init_cell_selection_weights_to_zero,
|
||||
reset_position_index_per_cell=self.reset_position_index_per_cell,
|
||||
disable_per_token_loss=self.disable_per_token_loss,
|
||||
return_dict=True,
|
||||
)
|
||||
|
||||
return (
|
||||
config,
|
||||
input_ids,
|
||||
input_mask,
|
||||
token_type_ids,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
label_ids,
|
||||
numeric_values,
|
||||
numeric_values_scale,
|
||||
float_answer,
|
||||
aggregation_labels,
|
||||
)
|
||||
|
||||
def create_and_check_model(
|
||||
self,
|
||||
config,
|
||||
input_ids,
|
||||
input_mask,
|
||||
token_type_ids,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
label_ids,
|
||||
numeric_values,
|
||||
numeric_values_scale,
|
||||
float_answer,
|
||||
aggregation_labels,
|
||||
):
|
||||
model = TapasModel(config=config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
result = model(input_ids, attention_mask=input_mask, token_type_ids=token_type_ids)
|
||||
result = model(input_ids, token_type_ids=token_type_ids)
|
||||
result = model(input_ids)
|
||||
self.parent.assertEqual(result.last_hidden_state.shape, (self.batch_size, self.seq_length, self.hidden_size))
|
||||
self.parent.assertEqual(result.pooler_output.shape, (self.batch_size, self.hidden_size))
|
||||
|
||||
# def create_and_check_for_masked_lm(
|
||||
# self,
|
||||
# config,
|
||||
# input_ids,
|
||||
# input_mask,
|
||||
# token_type_ids,
|
||||
# sequence_labels,
|
||||
# token_labels,
|
||||
# label_ids,
|
||||
# numeric_values,
|
||||
# numeric_values_scale,
|
||||
# float_answer,
|
||||
# aggregation_labels,
|
||||
# ):
|
||||
# model = TapasForMaskedLM(config=config)
|
||||
# model.to(torch_device)
|
||||
# model.eval()
|
||||
# result = model(input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, labels=token_labels)
|
||||
# self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length, self.vocab_size))
|
||||
|
||||
def create_and_check_for_question_answering(
|
||||
self,
|
||||
config,
|
||||
input_ids,
|
||||
input_mask,
|
||||
token_type_ids,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
label_ids,
|
||||
numeric_values,
|
||||
numeric_values_scale,
|
||||
float_answer,
|
||||
aggregation_labels,
|
||||
):
|
||||
# inference: without aggregation head (SQA).
|
||||
sqa_config = copy.copy(config)
|
||||
sqa_config.num_aggregation_labels = 0
|
||||
sqa_config.use_answer_as_supervision = False
|
||||
model = TapasForQuestionAnswering(config=sqa_config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
result = model(
|
||||
input_ids=input_ids,
|
||||
attention_mask=input_mask,
|
||||
token_type_ids=token_type_ids,
|
||||
)
|
||||
self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length))
|
||||
|
||||
# inference: with aggregation head (WTQ, WikiSQL-supervised)
|
||||
model = TapasForQuestionAnswering(config=config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
result = model(
|
||||
input_ids=input_ids,
|
||||
attention_mask=input_mask,
|
||||
token_type_ids=token_type_ids,
|
||||
)
|
||||
self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length))
|
||||
self.parent.assertEqual(result.logits_aggregation.shape, (self.batch_size, self.num_aggregation_labels))
|
||||
|
||||
# training: can happen in 3 main ways
|
||||
# case 1: conversational (SQA)
|
||||
model = TapasForQuestionAnswering(config=sqa_config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
result = model(
|
||||
input_ids,
|
||||
attention_mask=input_mask,
|
||||
token_type_ids=token_type_ids,
|
||||
label_ids=label_ids,
|
||||
)
|
||||
self.parent.assertEqual(result.loss.shape, ())
|
||||
self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length))
|
||||
|
||||
# case 2: weak supervision for aggregation (WTQ)
|
||||
model = TapasForQuestionAnswering(config=config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
result = model(
|
||||
input_ids=input_ids,
|
||||
attention_mask=input_mask,
|
||||
token_type_ids=token_type_ids,
|
||||
label_ids=label_ids,
|
||||
numeric_values=numeric_values,
|
||||
numeric_values_scale=numeric_values_scale,
|
||||
float_answer=float_answer,
|
||||
)
|
||||
self.parent.assertEqual(result.loss.shape, ())
|
||||
self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length))
|
||||
self.parent.assertEqual(result.logits_aggregation.shape, (self.batch_size, self.num_aggregation_labels))
|
||||
|
||||
# case 3: strong supervision for aggregation (WikiSQL-supervised)
|
||||
wikisql_config = copy.copy(config)
|
||||
wikisql_config.use_answer_as_supervision = False
|
||||
model = TapasForQuestionAnswering(config=wikisql_config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
result = model(
|
||||
input_ids,
|
||||
attention_mask=input_mask,
|
||||
token_type_ids=token_type_ids,
|
||||
label_ids=label_ids,
|
||||
aggregation_labels=aggregation_labels,
|
||||
)
|
||||
self.parent.assertEqual(result.loss.shape, ())
|
||||
self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length))
|
||||
self.parent.assertEqual(result.logits_aggregation.shape, (self.batch_size, self.num_aggregation_labels))
|
||||
|
||||
def create_and_check_for_sequence_classification(
|
||||
self,
|
||||
config,
|
||||
input_ids,
|
||||
input_mask,
|
||||
token_type_ids,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
label_ids,
|
||||
numeric_values,
|
||||
numeric_values_scale,
|
||||
float_answer,
|
||||
aggregation_labels,
|
||||
):
|
||||
config.num_labels = self.num_labels
|
||||
model = TapasForSequenceClassification(config)
|
||||
model.to(torch_device)
|
||||
model.eval()
|
||||
result = model(input_ids, attention_mask=input_mask, labels=sequence_labels)
|
||||
self.parent.assertEqual(result.logits.shape, (self.batch_size, self.num_labels))
|
||||
|
||||
def prepare_config_and_inputs_for_common(self):
|
||||
config_and_inputs = self.prepare_config_and_inputs()
|
||||
(
|
||||
config,
|
||||
input_ids,
|
||||
input_mask,
|
||||
token_type_ids,
|
||||
sequence_labels,
|
||||
token_labels,
|
||||
label_ids,
|
||||
numeric_values,
|
||||
numeric_values_scale,
|
||||
float_answer,
|
||||
aggregation_labels,
|
||||
) = config_and_inputs
|
||||
inputs_dict = {"input_ids": input_ids, "token_type_ids": token_type_ids, "attention_mask": input_mask}
|
||||
return config, inputs_dict
|
||||
|
||||
|
||||
@require_torch
|
||||
class TapasModelTest(ModelTesterMixin, unittest.TestCase):
|
||||
|
||||
all_model_classes = (
|
||||
(
|
||||
TapasModel,
|
||||
#TapasForMaskedLM,
|
||||
TapasForQuestionAnswering,
|
||||
TapasForSequenceClassification,
|
||||
)
|
||||
if is_torch_available()
|
||||
else None
|
||||
)
|
||||
test_pruning = False
|
||||
test_torchscript = True
|
||||
test_resize_embeddings = True
|
||||
test_head_masking = False
|
||||
|
||||
def setUp(self):
|
||||
self.model_tester = TapasModelTester(self)
|
||||
self.config_tester = ConfigTester(self, config_class=TapasConfig, dim=37)
|
||||
|
||||
def test_config(self):
|
||||
self.config_tester.run_common_tests()
|
||||
|
||||
def test_model(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_model(*config_and_inputs)
|
||||
|
||||
# def test_for_masked_lm(self):
|
||||
# config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
# self.model_tester.create_and_check_for_masked_lm(*config_and_inputs)
|
||||
|
||||
def test_for_question_answering(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_for_question_answering(*config_and_inputs)
|
||||
|
||||
def test_for_sequence_classification(self):
|
||||
config_and_inputs = self.model_tester.prepare_config_and_inputs()
|
||||
self.model_tester.create_and_check_for_sequence_classification(*config_and_inputs)
|
||||
|
||||
|
||||
def prepare_tapas_single_inputs_for_inference():
|
||||
# Here we prepare a single table-question pair to test TAPAS inference on:
|
||||
data = {'Footballer': ["Lionel Messi", "Cristiano Ronaldo"],
|
||||
'Age': ["33", "35"],
|
||||
}
|
||||
queries = "Which footballer is 33 years old?"
|
||||
table = pd.DataFrame.from_dict(data)
|
||||
|
||||
return table, queries
|
||||
|
||||
|
||||
def prepare_tapas_batch_inputs_for_inference():
|
||||
# Here we prepare a batch of 2 table-question pairs to test TAPAS inference on:
|
||||
data = {'Footballer': ["Lionel Messi", "Cristiano Ronaldo"],
|
||||
'Age': ["33", "35"],
|
||||
'Number of goals': ["712", "750"]
|
||||
}
|
||||
queries = ["Which footballer is 33 years old?", "How many goals does Ronaldo have?"]
|
||||
table = pd.DataFrame.from_dict(data)
|
||||
|
||||
return table, queries
|
||||
|
||||
|
||||
def prepare_tapas_batch_inputs_for_training():
|
||||
# Here we prepare a DIFFERENT batch of 2 table-question pairs to test TAPAS training on:
|
||||
data = {'Footballer': ["Lionel Messi", "Cristiano Ronaldo"],
|
||||
'Age': ["33", "35"],
|
||||
'Number of goals': ["712", "750"]
|
||||
}
|
||||
queries = ["Which footballer is 33 years old?", "What's the total number of goals?"]
|
||||
table = pd.DataFrame.from_dict(data)
|
||||
|
||||
answer_coordinates = [[(0, 0)], [(0, 2), (1, 2)]]
|
||||
answer_text = [["Lionel Messi"], ["1462"]]
|
||||
float_answer = [float("NaN"), float("1462")]
|
||||
|
||||
return table, queries, answer_coordinates, answer_text, float_answer
|
||||
|
||||
|
||||
@require_torch
|
||||
class TapasModelIntegrationTest(unittest.TestCase):
|
||||
@cached_property
|
||||
def default_tokenizer(self):
|
||||
return TapasTokenizer.from_pretrained("nielsr/tapas-base-finetuned-wtq")
|
||||
|
||||
@slow
|
||||
def test_inference_no_head(self):
|
||||
# ideally we want to test this with the weights of tapas_inter_masklm_base_reset,
|
||||
# but since it's not straightforward to do this with the TF 1 implementation, we test it with
|
||||
# the weights of the WTQ base model (i.e. tapas_wtq_wikisql_sqa_inter_masklm_base_reset)
|
||||
model = TapasModel.from_pretrained("nielsr/tapas-base-finetuned-wtq")
|
||||
|
||||
tokenizer = default_tokenizer()
|
||||
table, queries = prepare_tapas_single_inputs_for_inference()
|
||||
inputs = tokenizer(table=table, queries=queries, return_tensors="pt")
|
||||
outputs = model(**inputs)
|
||||
# test the sequence output
|
||||
expected_slice = torch.tensor(
|
||||
[[[-0.141581565, -0.599805772, 0.747186482],
|
||||
[-0.143664181, -0.602008104, 0.749218345],
|
||||
[-0.15169853, -0.603363097, 0.741370678]]]
|
||||
)
|
||||
|
||||
self.assertTrue(torch.allclose(outputs.sequence_output[:, :3, :3], expected_slice, atol=1e-4))
|
||||
|
||||
# test the pooled output
|
||||
expected_slice = torch.tensor(
|
||||
[[0.987518311, -0.970520139, -0.994303405]]
|
||||
)
|
||||
|
||||
self.assertTrue(torch.allclose(outputs.pooled_output[:, :3], expected_slice, atol=1e-4))
|
||||
|
||||
|
||||
@unittest.skip(reason="Model not available yet")
|
||||
def test_inference_masked_lm(self):
|
||||
pass
|
||||
|
||||
# TapasForQuestionAnswering has 3 possible ways of being fine-tuned:
|
||||
# - conversational set-up (SQA)
|
||||
# - weak supervision for aggregation (WTQ, WikiSQL)
|
||||
# - strong supervision for aggregation (WikiSQL-supervised)
|
||||
# We test all of them:
|
||||
@slow
|
||||
def test_inference_question_answering_head_conversational(self):
|
||||
# note that nielsr/tapas-base-finetuned-sqa should correspond to tapas_sqa_inter_masklm_base_reset
|
||||
model = TapasForQuestionAnswering.from_pretrained("nielsr/tapas-base-finetuned-sqa")
|
||||
|
||||
tokenizer = default_tokenizer()
|
||||
table, queries = prepare_tapas_single_inputs_for_inference()
|
||||
inputs = tokenizer(table=table, queries=queries, return_tensors="pt")
|
||||
outputs = model(**inputs)
|
||||
# test the logits
|
||||
logits = outputs.logits
|
||||
expected_shape = torch.Size((1, 21))
|
||||
self.assertEqual(logits.shape, expected_shape)
|
||||
expected_tensor = torch.tensor([[-9997.22461, -9997.22461, -9997.22461, -9997.22461, -9997.22461,
|
||||
-9997.22461, -9997.22461, -9997.22461, -9997.22461, -16.2628059,
|
||||
-10004.082, 15.4330549, 15.4330549, 15.4330549, -9990.42,
|
||||
-16.3270779, -16.3270779, -16.3270779, -16.3270779, -16.3270779, -10004.8506]]) # ok
|
||||
|
||||
self.assertTrue(torch.allclose(logits, expected_tensor, atol=1e-4))
|
||||
|
||||
@slow
|
||||
def test_inference_question_answering_head_weak_supervision(self):
|
||||
# note that nielsr/tapas-base-finetuned-wtq should correspond to tapas_wtq_wikisql_sqa_inter_masklm_base_reset
|
||||
model = TapasForQuestionAnswering.from_pretrained("nielsr/tapas-base-finetuned-wtq")
|
||||
|
||||
tokenizer = default_tokenizer()
|
||||
# let's test on a batch
|
||||
table, queries = prepare_tapas_batch_inputs_for_inference()
|
||||
inputs = tokenizer(table=table, queries=queries, padding="longest", return_tensors="pt")
|
||||
outputs = model(**inputs)
|
||||
# test the logits
|
||||
logits = outputs.logits
|
||||
expected_shape = torch.Size((2, 28))
|
||||
self.assertEqual(logits.shape, expected_shape)
|
||||
expected_slice = torch.tensor([[-160.375504, -160.375504, -160.375504, -10072.3965, -10070.9414, -10094.9736],
|
||||
[-9861.6123, -9861.6123, -9861.6123, -9861.6123, -9891.01172, 146.600677]]) # ok (batch size = 2)
|
||||
|
||||
self.assertTrue(torch.allclose(logits[:,-6:], expected_slice, atol=1e-4))
|
||||
|
||||
# test the aggregation logits
|
||||
logits_aggregation = outputs.logits_aggregation
|
||||
expected_shape = torch.Size((2, 4))
|
||||
self.assertEqual(logits_aggregation.shape, expected_shape)
|
||||
expected_tensor = torch.tensor([[18.8545208, -9.76614857, -6.3128891, -2.93525243],
|
||||
[-4.05782509, 40.0351, -5.35329962, 23.3978653]]) # ok (batch size = 2)
|
||||
|
||||
self.assertTrue(torch.allclose(logits_aggregation, expected_tensor, atol=1e-4))
|
||||
|
||||
tokenizer = default_tokenizer()
|
||||
|
||||
@slow
|
||||
def test_training_question_answering_head_weak_supervision(self):
|
||||
# note that nielsr/tapas-base-finetuned-wtq should correspond to tapas_wtq_wikisql_sqa_inter_masklm_base_reset
|
||||
model = TapasForQuestionAnswering.from_pretrained("nielsr/tapas-base-finetuned-wtq")
|
||||
model.to(torch_device)
|
||||
|
||||
tokenizer = default_tokenizer()
|
||||
# let's test on a batch
|
||||
table, queries, answer_coordinates, answer_text, float_answer = prepare_tapas_batch_inputs_for_training()
|
||||
inputs = tokenizer(table=table, queries=queries, answer_coordinates=answer_coordinates,
|
||||
answer_text=answer_text, padding="longest", return_tensors="pt")
|
||||
|
||||
# prepare data (created by the tokenizer) and move to torch_device
|
||||
input_ids = inputs["input_ids"].to(torch_device)
|
||||
attention_mask = inputs["attention_mask"].to(torch_device)
|
||||
token_type_ids = inputs["token_type_ids"].to(torch_device)
|
||||
label_ids = inputs["label_ids"].to(torch_device)
|
||||
numeric_values = inputs["numeric_values"].to(torch_device)
|
||||
numeric_values_scale = inputs["numeric_values_scale"].to(torch_device)
|
||||
|
||||
# the answer should be prepared by the user
|
||||
float_answer = torch.FloatTensor(float_answer).to(torch_device)
|
||||
|
||||
# forward pass to get loss + logits:
|
||||
outputs = model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids, label_ids=label_ids,
|
||||
numeric_values=numeric_values, numeric_values_scale=numeric_values_scale,
|
||||
float_answer=float_answer)
|
||||
|
||||
# test the loss
|
||||
loss = outputs.loss
|
||||
expected_loss = 3.3527612686157227e-08 # ok
|
||||
self.assertEqual(loss.item(), expected_loss, atol=1e-4)
|
||||
|
||||
# test the logits on the first example
|
||||
logits = outputs.logits
|
||||
expected_shape = torch.Size((2, 28))
|
||||
self.assertEqual(logits.shape, expected_shape)
|
||||
expected_slice = torch.tensor([-160.0156, -160.0156, -160.0156, -160.0156, -160.0156,
|
||||
-10072.2266, -10070.8896, -10092.6006, -10092.6006]) # ok
|
||||
|
||||
self.assertTrue(torch.allclose(logits[:,-9:], expected_slice, atol=1e-4))
|
||||
|
||||
|
||||
# test the aggregation logits on the second example
|
||||
logits_aggregation = outputs.logits_aggregation
|
||||
expected_shape = torch.Size((2, 4))
|
||||
self.assertEqual(logits_aggregation.shape, expected_shape)
|
||||
expected_slice = torch.tensor([-4.0538, 40.0304, -5.3554, 23.3965]) # ok
|
||||
|
||||
self.assertTrue(torch.allclose(logits_aggregation[1,-4:], expected_slice, atol=1e-4))
|
||||
|
||||
@slow
|
||||
def test_inference_question_answering_head_strong_supervision(self):
|
||||
# note that nielsr/tapas-base-finetuned-wikisql-supervised should correspond to tapas_wikisql_sqa_inter_masklm_base_reset
|
||||
model = TapasForQuestionAnswering.from_pretrained("nielsr/tapas-base-finetuned-wikisql-supervised")
|
||||
|
||||
tokenizer = default_tokenizer()
|
||||
table, queries = prepare_tapas_single_inputs_for_inference()
|
||||
inputs = tokenizer(table=table, queries=queries, return_tensors="pt")
|
||||
outputs = model(**inputs)
|
||||
# test the logits
|
||||
logits = outputs.logits
|
||||
expected_shape = torch.Size((1, 21))
|
||||
self.assertEqual(logits.shape, expected_shape)
|
||||
expected_tensor = torch.tensor([[-10011.1084, -10011.1084, -10011.1084, -10011.1084, -10011.1084,
|
||||
-10011.1084, -10011.1084, -10011.1084, -10011.1084, -18.6185989,
|
||||
-10008.7969, 17.6355762, 17.6355762, 17.6355762, -10002.4404,
|
||||
-18.7111301, -18.7111301, -18.7111301, -18.7111301, -18.7111301, -10007.0977]]) # ok
|
||||
|
||||
self.assertTrue(torch.allclose(logits, expected_tensor, atol=1e-4))
|
||||
|
||||
# test the aggregation logits
|
||||
logits_aggregation = outputs.logits_aggregation
|
||||
expected_shape = torch.Size((1, 4))
|
||||
self.assertEqual(logits_aggregation.shape, expected_shape)
|
||||
expected_tensor = torch.tensor([[16.5659733, -3.06624889, -2.34152961, -0.970244825]]) # ok, PyTorch model outputs [[16.5679, -3.0668, -2.3442, -0.9674]]
|
||||
|
||||
self.assertTrue(torch.allclose(logits_aggregation, expected_tensor, atol=1e-4))
|
||||
|
||||
@slow
|
||||
def test_inference_classification_head(self):
|
||||
# note that nielsr/tapas-base-finetuned-tabfact should correspond to tapas_tabfact_inter_masklm_base_reset
|
||||
model = TapasForSequenceClassification.from_pretrained("nielsr/tapas-base-finetuned-tabfact")
|
||||
|
||||
inputs = prepare_tapas_inputs_for_inference()
|
||||
outputs = model(**inputs)
|
||||
|
||||
# test the classification logits
|
||||
logits = outputs.logits
|
||||
expected_shape = torch.Size((1, 2))
|
||||
self.assertEqual(logits.shape, expected_shape)
|
||||
expected_tensor = torch.tensor([[0.795137286, 9.5572]]) # ok. Note that the PyTorch model outputs [[0.8057, 9.5281]]
|
||||
|
||||
self.assertTrue(torch.allclose(outputs.logits, expected_tensor, atol=1e-4))
|
||||
|
||||
# Below: tests for Tapas utilities which are defined in modeling_tapas.py.
|
||||
# These are based on segmented_tensor_test.py of the original implementation.
|
||||
# URL: https://github.com/google-research/tapas/blob/master/tapas/models/segmented_tensor_test.py
|
||||
class TapasUtilitiesTest(unittest.TestCase):
|
||||
def _prepare_tables(self):
|
||||
"""Prepares two tables, both with three distinct rows.
|
||||
The first table has two columns:
|
||||
1.0, 2.0 | 3.0
|
||||
2.0, 0.0 | 1.0
|
||||
1.0, 3.0 | 4.0
|
||||
The second table has three columns:
|
||||
1.0 | 2.0 | 3.0
|
||||
2.0 | 0.0 | 1.0
|
||||
1.0 | 3.0 | 4.0
|
||||
Returns:
|
||||
SegmentedTensors with the tables.
|
||||
"""
|
||||
values = torch.tensor(
|
||||
[
|
||||
[[1.0, 2.0, 3.0], [2.0, 0.0, 1.0], [1.0, 3.0, 4.0]],
|
||||
[[1.0, 2.0, 3.0], [2.0, 0.0, 1.0], [1.0, 3.0, 4.0]],
|
||||
]
|
||||
)
|
||||
row_index = IndexMap(
|
||||
indices=torch.tensor(
|
||||
[
|
||||
[[0, 0, 0], [1, 1, 1], [2, 2, 2]],
|
||||
[[0, 0, 0], [1, 1, 1], [2, 2, 2]],
|
||||
]
|
||||
),
|
||||
num_segments=3,
|
||||
batch_dims=1,
|
||||
)
|
||||
col_index = IndexMap(
|
||||
indices=torch.tensor(
|
||||
[
|
||||
[[0, 0, 1], [0, 0, 1], [0, 0, 1]],
|
||||
[[0, 1, 2], [0, 1, 2], [0, 1, 2]],
|
||||
]
|
||||
),
|
||||
num_segments=3,
|
||||
batch_dims=1,
|
||||
)
|
||||
return values, row_index, col_index
|
||||
|
||||
def test_product_index(self):
|
||||
_, row_index, col_index = self._prepare_tables()
|
||||
cell_index = ProductIndexMap(row_index, col_index)
|
||||
row_index_proj = cell_index.project_outer(cell_index)
|
||||
col_index_proj = cell_index.project_inner(cell_index)
|
||||
|
||||
ind = cell_index.indices
|
||||
self.assertEqual(cell_index.num_segments, 9)
|
||||
|
||||
# Projections should give back the original indices.
|
||||
# we use np.testing.assert_array_equal rather than Tensorflow's assertAllEqual
|
||||
np.testing.assert_array_equal(row_index.indices.numpy(), row_index_proj.indices.numpy())
|
||||
self.assertEqual(row_index.num_segments, row_index_proj.num_segments)
|
||||
self.assertEqual(row_index.batch_dims, row_index_proj.batch_dims)
|
||||
# We use np.testing.assert_array_equal rather than Tensorflow's assertAllEqual
|
||||
np.testing.assert_array_equal(col_index.indices.numpy(), col_index_proj.indices.numpy())
|
||||
self.assertEqual(col_index.batch_dims, col_index_proj.batch_dims)
|
||||
|
||||
# The first and second "column" are identified in the first table.
|
||||
for i in range(3):
|
||||
self.assertEqual(ind[0, i, 0], ind[0, i, 1])
|
||||
self.assertNotEqual(ind[0, i, 0], ind[0, i, 2])
|
||||
|
||||
# All rows are distinct in the first table.
|
||||
for i, i_2 in zip(range(3), range(3)):
|
||||
for j, j_2 in zip(range(3), range(3)):
|
||||
if i != i_2 and j != j_2:
|
||||
self.assertNotEqual(ind[0, i, j], ind[0, i_2, j_2])
|
||||
|
||||
# All cells are distinct in the second table.
|
||||
for i, i_2 in zip(range(3), range(3)):
|
||||
for j, j_2 in zip(range(3), range(3)):
|
||||
if i != i_2 or j != j_2:
|
||||
self.assertNotEqual(ind[1, i, j], ind[1, i_2, j_2])
|
||||
|
||||
def test_flatten(self):
|
||||
_, row_index, col_index = self._prepare_tables()
|
||||
row_index_flat = flatten(row_index)
|
||||
col_index_flat = flatten(col_index)
|
||||
|
||||
shape = [3, 4, 5]
|
||||
batched_index = IndexMap(indices=torch.zeros(shape).type(torch.LongTensor), num_segments=1, batch_dims=3)
|
||||
batched_index_flat = flatten(batched_index)
|
||||
|
||||
# We use np.testing.assert_array_equal rather than Tensorflow's assertAllEqual
|
||||
np.testing.assert_array_equal(
|
||||
row_index_flat.indices.numpy(), [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5]
|
||||
)
|
||||
np.testing.assert_array_equal(
|
||||
col_index_flat.indices.numpy(), [0, 0, 1, 0, 0, 1, 0, 0, 1, 3, 4, 5, 3, 4, 5, 3, 4, 5]
|
||||
)
|
||||
self.assertEqual(batched_index_flat.num_segments.numpy(), np.prod(shape))
|
||||
np.testing.assert_array_equal(batched_index_flat.indices.numpy(), range(np.prod(shape)))
|
||||
|
||||
def test_range_index_map(self):
|
||||
batch_shape = [3, 4]
|
||||
num_segments = 5
|
||||
index = range_index_map(batch_shape, num_segments)
|
||||
|
||||
self.assertEqual(num_segments, index.num_segments)
|
||||
self.assertEqual(2, index.batch_dims)
|
||||
indices = index.indices
|
||||
# We use np.testing.assert_array_equal rather than Tensorflow's assertAllEqual
|
||||
np.testing.assert_array_equal(list(indices.size()), [3, 4, 5])
|
||||
for i in range(batch_shape[0]):
|
||||
for j in range(batch_shape[1]):
|
||||
# We use np.testing.assert_array_equal rather than Tensorflow's assertAllEqual
|
||||
np.testing.assert_array_equal(indices[i, j, :].numpy(), range(num_segments))
|
||||
|
||||
def test_reduce_sum(self):
|
||||
values, row_index, col_index = self._prepare_tables()
|
||||
cell_index = ProductIndexMap(row_index, col_index)
|
||||
row_sum, _ = reduce_sum(values, row_index)
|
||||
col_sum, _ = reduce_sum(values, col_index)
|
||||
cell_sum, _ = reduce_sum(values, cell_index)
|
||||
|
||||
# We use np.testing.assert_allclose rather than Tensorflow's assertAllClose
|
||||
np.testing.assert_allclose(row_sum.numpy(), [[6.0, 3.0, 8.0], [6.0, 3.0, 8.0]])
|
||||
np.testing.assert_allclose(col_sum.numpy(), [[9.0, 8.0, 0.0], [4.0, 5.0, 8.0]])
|
||||
np.testing.assert_allclose(
|
||||
cell_sum.numpy(),
|
||||
[[3.0, 3.0, 0.0, 2.0, 1.0, 0.0, 4.0, 4.0, 0.0], [1.0, 2.0, 3.0, 2.0, 0.0, 1.0, 1.0, 3.0, 4.0]],
|
||||
)
|
||||
|
||||
def test_reduce_mean(self):
|
||||
values, row_index, col_index = self._prepare_tables()
|
||||
cell_index = ProductIndexMap(row_index, col_index)
|
||||
row_mean, _ = reduce_mean(values, row_index)
|
||||
col_mean, _ = reduce_mean(values, col_index)
|
||||
cell_mean, _ = reduce_mean(values, cell_index)
|
||||
|
||||
# We use np.testing.assert_allclose rather than Tensorflow's assertAllClose
|
||||
np.testing.assert_allclose(
|
||||
row_mean.numpy(), [[6.0 / 3.0, 3.0 / 3.0, 8.0 / 3.0], [6.0 / 3.0, 3.0 / 3.0, 8.0 / 3.0]]
|
||||
)
|
||||
np.testing.assert_allclose(col_mean.numpy(), [[9.0 / 6.0, 8.0 / 3.0, 0.0], [4.0 / 3.0, 5.0 / 3.0, 8.0 / 3.0]])
|
||||
np.testing.assert_allclose(
|
||||
cell_mean.numpy(),
|
||||
[
|
||||
[3.0 / 2.0, 3.0, 0.0, 2.0 / 2.0, 1.0, 0.0, 4.0 / 2.0, 4.0, 0.0],
|
||||
[1.0, 2.0, 3.0, 2.0, 0.0, 1.0, 1.0, 3.0, 4.0],
|
||||
],
|
||||
)
|
||||
|
||||
def test_reduce_max(self):
|
||||
values = torch.as_tensor([2.0, 1.0, 0.0, 3.0])
|
||||
index = IndexMap(indices=torch.as_tensor([0, 1, 0, 1]), num_segments=2)
|
||||
maximum, _ = reduce_max(values, index)
|
||||
|
||||
# We use np.testing.assert_array_equal rather than Tensorflow's assertAllEqual
|
||||
np.testing.assert_array_equal(maximum.numpy(), [2, 3])
|
||||
|
||||
def test_reduce_sum_vectorized(self):
|
||||
values = torch.as_tensor([[1.0, 2.0, 3.0], [2.0, 3.0, 4.0], [3.0, 4.0, 5.0]])
|
||||
index = IndexMap(indices=torch.as_tensor([0, 0, 1]), num_segments=2, batch_dims=0)
|
||||
sums, new_index = reduce_sum(values, index)
|
||||
|
||||
# We use np.testing.assert_allclose rather than Tensorflow's assertAllClose
|
||||
np.testing.assert_allclose(sums.numpy(), [[3.0, 5.0, 7.0], [3.0, 4.0, 5.0]])
|
||||
# We use np.testing.assert_array_equal rather than Tensorflow's assertAllEqual
|
||||
np.testing.assert_array_equal(new_index.indices.numpy(), [0, 1])
|
||||
np.testing.assert_array_equal(new_index.num_segments.numpy(), 2)
|
||||
np.testing.assert_array_equal(new_index.batch_dims, 0)
|
||||
|
||||
def test_gather(self):
|
||||
values, row_index, col_index = self._prepare_tables()
|
||||
cell_index = ProductIndexMap(row_index, col_index)
|
||||
|
||||
# Compute sums and then gather. The result should have the same shape as
|
||||
# the original table and each element should contain the sum the values in
|
||||
# its cell.
|
||||
sums, _ = reduce_sum(values, cell_index)
|
||||
cell_sum = gather(sums, cell_index)
|
||||
assert cell_sum.size() == values.size()
|
||||
|
||||
# We use np.testing.assert_array_equal rather than Tensorflow's assertAllEqual
|
||||
np.testing.assert_allclose(
|
||||
cell_sum.numpy(),
|
||||
[[[3.0, 3.0, 3.0], [2.0, 2.0, 1.0], [4.0, 4.0, 4.0]], [[1.0, 2.0, 3.0], [2.0, 0.0, 1.0], [1.0, 3.0, 4.0]]],
|
||||
)
|
||||
|
||||
def test_gather_vectorized(self):
|
||||
values = torch.as_tensor([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])
|
||||
index = IndexMap(indices=torch.as_tensor([[0, 1], [1, 0]]), num_segments=2, batch_dims=1)
|
||||
result = gather(values, index)
|
||||
|
||||
# We use np.testing.assert_array_equal rather than Tensorflow's assertAllEqual
|
||||
np.testing.assert_array_equal(result.numpy(), [[[1, 2], [3, 4]], [[7, 8], [5, 6]]])
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user