c-my/transformers
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Incorporate suggestions by Google authors

Browse Source
This commit is contained in:
Rogge Niels
2020-12-09 21:18:21 +01:00
parent 4cb0339945
commit 9fa0881bc2
3 changed files with 31 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/model_doc/tapas.rst
+21 -17
View File
@@ -5,8 +5,8 @@ Overview
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The TAPAS model was proposed in `TAPAS: Weakly Supervised Table Parsing via Pre-training
<https://arxiv.org/abs/2004.02349>`__ by Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno and
Julian Martin Eisenschlos. It's a BERT-based model specifically designed (and pre-trained) for answering questions
<https://www.aclweb.org/anthology/2020.acl-main.398>`__ by Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco
Piccinno and Julian Martin Eisenschlos. It's a BERT-based model specifically designed (and pre-trained) for answering questions
about tabular data. Compared to BERT, TAPAS uses relative position embeddings and has 7 token types that encode tabular
structure. TAPAS is pre-trained on the masked language modeling (MLM) objective on a large dataset comprising millions
of tables from English Wikipedia and corresponding texts. For question answering, TAPAS has 2 heads on top: a cell
@@ -39,16 +39,20 @@ table entailment (a binary classification task). For more details, see their fol
intermediate pre-training <https://arxiv.org/abs/2010.00571>`__ by Julian Martin Eisenschlos, Syrine Krichene and
Thomas Müller.
The original code can be found `here <https://github.com/google-research/tapas>`__.
The original code can be found `here <https://github.com/google-research/tapas>`__. This page also includes links to the
datasets (SQA, WTQ, WikiSQL and TabFact).
Tips:
- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell
of the table). According to the authors, this usually results in a slightly better performance, and allows you to
encode longer sequences without running out of embeddings. This is reflected in the ``reset_position_index_per_cell``
parameter of :class:`~transformers.TapasConfig`, which is set to ``True`` by default.
There are both pre-trained models in the `model hub <https://huggingface.co/models?search=tapas>`_ with absolute and relative
position embeddings. Note that it's usually advised to pad the inputs on the right rather than the left.
of the table). Note that this is something that was added after the publication of the original TAPAS paper. According
to the authors, this usually results in a slightly better performance, and allows you to encode longer sequences without
running out of embeddings. This is reflected in the ``reset_position_index_per_cell`` parameter of :class:`~transformers.TapasConfig`,
which is set to ``True`` by default.
The latest versions of the models available in the `model hub <https://huggingface.co/models?search=tapas>`_ all use relative
position embeddings. You can still use the ones with absolute position embeddings by passing in a certain version when calling the
`.from_pretrained` method as explained in the model cards.
Note that it's usually advised to pad the inputs on the right rather than the left.
- TAPAS is based on BERT, so ``TAPAS-base`` for example corresponds to a ``BERT-base`` architecture. Of course, TAPAS-large
will result in the best performance (the results reported in the paper are from TAPAS-large). Metrics of the various
sized models are shown on the `original Github repository <https://github.com/google-research/tapas>`_.
@@ -77,13 +81,13 @@ the different datasets on which Tapas was fine-tuned:
1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you
first ask "what's the name of the first actor?" then you can ask a follow-up question such as "how old is he?". Here, questions
do not involve any aggregation (all questions are cell selection questions).
2. WTQ/WikiSQL: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related
2. WTQ: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related
to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values.
You can then for example ask "what's the total number of goals Cristiano Ronaldo made in his career?". This case is also called **weak
supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer
to the question as supervision.
3. WikiSQL-supervised: this dataset is actually the same dataset as WikiSQL, but here the model is given the ground truth aggregation
operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier.
3. WikiSQL-supervised: this dataset is based on WikiSQL with the model being given the ground truth aggregation operator during training.
This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier.
To summarize:
@@ -92,7 +96,7 @@ To summarize:
+------------------------------------+----------------------+-------------------------------------------------------------------------------------------------------------------+
| Conversational | SQA | Conversational, only cell selection questions |
+------------------------------------+----------------------+-------------------------------------------------------------------------------------------------------------------+
| Weak supervision for aggregation | WTQ, WikiSQL | Questions might involve aggregation, and the model must learn this given only the answer as supervision |
| Weak supervision for aggregation | WTQ | Questions might involve aggregation, and the model must learn this given only the answer as supervision |
+------------------------------------+----------------------+-------------------------------------------------------------------------------------------------------------------+
| Strong supervision for aggregation | WikiSQL-supervised | Questions might involve aggregation, and the model must learn this given the gold aggregation operator |
+------------------------------------+----------------------+-------------------------------------------------------------------------------------------------------------------+
@@ -107,7 +111,7 @@ Initializing a model with a pre-trained base and randomly initialized classifica
>>> model = TapasForQuestionAnswering.from_pretrained('google/tapas-base-uncased')
Of course, you don't necessarily have to follow one these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters
Of course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters
you want when initializing :class:`~transformers.TapasConfig`, and then create a :class:`~transformers.TapasForQuestionAnswering` based on that
configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it
this way. Here's an example:
@@ -136,7 +140,7 @@ This format is a TSV/CSV file with the following columns:
- ``id``: optional, id of the table-question pair, for bookkeeping purposes.
- ``annotator``: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.
- ``position``: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA).
You don't need this column in case you're going for WTQ/WikiSQL/WikiSQL-supervised.
You don't need this column in case you're going for WTQ/WikiSQL-supervised.
- ``question``: string
- ``table_file``: string, name of a csv file containing the tabular data
- ``answer_coordinates``: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)
@@ -145,7 +149,7 @@ This format is a TSV/CSV file with the following columns:
- ``float_answer``: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)
The tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion
scripts with some automated logic to convert the other datasets (WTQ and WikiSQL) into the SQA format. The author explains this `here <https://github.com/google-research/tapas/issues/50#issuecomment-705465960>`__.
scripts with some automated logic to convert the other datasets (WTQ, WikiSQL) into the SQA format. The author explains this `here <https://github.com/google-research/tapas/issues/50#issuecomment-705465960>`__.
Interestingly, these conversion scripts are not perfect (the ``answer_coordinates`` and ``float_answer`` fields are populated based on the ``answer_text``),
meaning that WTQ and WikiSQL results could actually be improved.
@@ -267,7 +271,7 @@ Usage: inference
Here we explain how you can use :class:`~transformers.TapasForQuestionAnswering` for inference (i.e. making predictions on new data).
For inference, only ``input_ids``, ``attention_mask`` and ``token_type_ids`` (which you can obtain using
:class:`~transformers.TapasTokenizer`) have to provided to the model to obtain the logits. Next, you can use the handy
:class:`~transformers.TapasTokenizer`) have to be provided to the model to obtain the logits. Next, you can use the handy
``convert_logits_to_predictions`` method of :class:`~transformers.TapasTokenizer` to convert these into predicted coordinates
and optional aggregation indices.
@@ -332,7 +336,7 @@ the ``prev_label_ids`` token types can be overwritten by the predicted ``label_i
Tapas specific outputs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: transformers.modeling_tapas.TableQuestionAnsweringOutput
.. autoclass:: transformers.models.tapas.modeling_tapas.TableQuestionAnsweringOutput
:members:
src/transformers/models/tapas/modeling_tapas.py
+8 -8
View File
@@ -1011,9 +1011,9 @@ class TapasLMHead(nn.Module):
@add_start_docstrings(
"""
Tapas Model with a cell selection head and optionally aggregation head on top for question-answering tasks on
tables (linear layers on top of the hidden-states output to compute `logits` and optionally `logits_aggregation`),
e.g. for SQA, WTQ or WikiSQL tasks.
Tapas Model with a cell selection head and optional aggregation head on top for question-answering tasks on
tables (linear layers on top of the hidden-states output to compute `logits` and optional `logits_aggregation`),
e.g. for SQA, WTQ or WikiSQL-supervised tasks.
""",
TAPAS_START_DOCSTRING,
)
@@ -1085,17 +1085,17 @@ class TapasForQuestionAnswering(TapasPreTrainedModel):
aggregation_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, )`, `optional`):
Aggregation function index for every example in the batch for computing the aggregation loss. Indices
should be in :obj:`[0, ..., config.num_aggregation_labels - 1]`. Only required in case of strong
supervision for aggregation (WikiSQL-SUPERVISED).
supervision for aggregation (WikiSQL-supervised).
float_answer (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, )`, `optional`):
Float answer for every example in the batch. Set to `float('nan')` for cell selection questions.
Only required in case of weak supervision (WTQ, WikiSQL) to calculate the aggregate mask and regression loss.
Only required in case of weak supervision (WTQ) to calculate the aggregate mask and regression loss.
numeric_values (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, seq_length)`, `optional`):
Numeric values of every token, NaN for tokens which are not numeric values. Can be obtained using
:class:`~transformers.TapasTokenizer`. Only required in case of weak supervision for aggregation (WTQ,
WikiSQL) to calculate the regression loss.
:class:`~transformers.TapasTokenizer`. Only required in case of weak supervision for aggregation (WTQ)
to calculate the regression loss.
numeric_values_scale (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, seq_length)`, `optional`):
Scale of the numeric values of every token. Can be obtained using :class:`~transformers.TapasTokenizer`.
Only required in case of weak supervision for aggregation (WTQ, WikiSQL) to calculate the regression loss.
Only required in case of weak supervision for aggregation (WTQ) to calculate the regression loss.
Returns:
tests/test_modeling_tapas.py
+2
View File
@@ -566,6 +566,7 @@ class TapasModelIntegrationTest(unittest.TestCase):
logits = outputs.logits
expected_shape = torch.Size((2, 28))
self.assertEqual(logits.shape, expected_shape)
expected_slice = torch.tensor([[-160.375504, -160.375504, -160.375504, -10072.3965, -10070.9414, -10094.9736],
[-9861.6123, -9861.6123, -9861.6123, -9861.6123, -9891.01172, 146.600677]],
device=torch_device) # ok (batch size = 2)
@@ -599,6 +600,7 @@ class TapasModelIntegrationTest(unittest.TestCase):
# note that nielsr/tapas-base-finetuned-wtq should correspond to tapas_wtq_wikisql_sqa_inter_masklm_base_reset
model = TapasForQuestionAnswering.from_pretrained("nielsr/tapas-base-finetuned-wtq").to(torch_device)
model.to(torch_device)
# normally we should put the model in training mode but it's a pain to do this with the TF 1 implementation
tokenizer = self.default_tokenizer
# let's test on a batch