Pass a few more tests

docs/source/model_doc/tapas.rst

@@ -6,63 +6,62 @@ Overview
 
 The TAPAS model was proposed in `TAPAS: Weakly Supervised Table Parsing via Pre-training
 <https://arxiv.org/abs/2004.02349>`__ by Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno and
-Julian Martin Eisenschlos.
-It's a BERT-based model specifically designed (and pre-trained) for answering questions about tabular data. Compared to 
-BERT, TAPAS uses relative position embeddings and has 7 token types that encode tabular structure. TAPAS is pre-trained 
-on the masked language modeling (MLM) objective on a large dataset comprising millions of tables from English Wikipedia and 
-corresponding texts. For question answering, TAPAS has 2 heads on top: a cell selection head and an aggregation head, for 
-(optionally) performing aggregations (such as counting or summing) among selected cells. TAPAS has been fine-tuned on several 
-datasets: SQA (Sequential Question Answering by Microsoft), WTQ (Wiki Table Questions by Stanford University) and WikiSQL 
-(by Salesforce). It achieves state-of-the-art on both SQA and WTQ, while having comparable performance to SOTA on WikiSQL, 
-with a much simpler architecture. 
+Julian Martin Eisenschlos. It's a BERT-based model specifically designed (and pre-trained) for answering questions
+about tabular data. Compared to BERT, TAPAS uses relative position embeddings and has 7 token types that encode tabular
+structure. TAPAS is pre-trained on the masked language modeling (MLM) objective on a large dataset comprising millions
+of tables from English Wikipedia and corresponding texts. For question answering, TAPAS has 2 heads on top: a cell
+selection head and an aggregation head, for (optionally) performing aggregations (such as counting or summing) among
+selected cells. TAPAS has been fine-tuned on several datasets: SQA (Sequential Question Answering by Microsoft), WTQ
+(Wiki Table Questions by Stanford University) and WikiSQL (by Salesforce). It achieves state-of-the-art on both SQA and
+WTQ, while having comparable performance to SOTA on WikiSQL, with a much simpler architecture.
 
 The abstract from the paper is the following:
 
-*Answering natural language questions over tables is usually seen as a semantic parsing task. 
-To alleviate the collection cost of full logical forms, one popular approach focuses on weak 
-supervision consisting of denotations instead of logical forms. However, training semantic parsers 
-from weak supervision poses difficulties, and in addition, the generated logical forms are only used
-as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an 
-approach to question answering over tables without generating logical forms. TAPAS trains from weak 
-supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding 
-aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, 
-initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, 
-and is trained end-to-end. We experiment with three different semantic parsing datasets, and find 
-that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on 
-SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but 
-with a simpler model architecture. We additionally find that transfer learning, which is trivial 
-in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.*
+*Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the
+collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations
+instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition,
+the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we
+present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak
+supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation
+operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective
+joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with
+three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by
+improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL
+and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our
+setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.*
 
-In addition, the authors have further pre-trained TAPAS to recognize table entailment, by creating a balanced dataset of millions 
-of automatically created training examples which are learned in an intermediate step prior to fine-tuning. The authors of TAPAS 
-call this further pre-training intermediate pre-training (since TAPAS is first pre-trained on MLM, and then on another dataset). 
-They found that intermediate pre-training further improves performance on SQA, achieving a new state-of-the-art as well as 
-state-of-the-art on TabFact, a large-scale dataset with 16k Wikipedia tables for table entailment (a binary classification task).
-For more details, see their new paper: `Understanding tables with intermediate pre-training <https://arxiv.org/abs/2010.00571>`__ 
-by Julian Martin Eisenschlos, Syrine Krichene and Thomas Müller.
+In addition, the authors have further pre-trained TAPAS to recognize table entailment, by creating a balanced dataset
+of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning.
+The authors of TAPAS call this further pre-training intermediate pre-training (since TAPAS is first pre-trained on MLM,
+and then on another dataset). They found that intermediate pre-training further improves performance on SQA, achieving
+a new state-of-the-art as well as state-of-the-art on TabFact, a large-scale dataset with 16k Wikipedia tables for
+table entailment (a binary classification task). For more details, see their new paper: `Understanding tables with
+intermediate pre-training <https://arxiv.org/abs/2010.00571>`__ by Julian Martin Eisenschlos, Syrine Krichene and
+Thomas Müller.
 
 The original code can be found `here <https://github.com/google-research/tapas>`__.
 
 Tips:
 
-- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell of the table). According to
-  the authors, this usually results in a slightly better performance, and allows you to encode longer sequences without running out 
-  of embeddings.
-  If you don't want this, you can set the `reset_position_index_per_cell` parameter of :class:`~transformers.TapasConfig` to False.
-- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a conversational set-up. This 
-  means that you can ask follow-up questions such as "what is his age?" related to the previous question. Note that the forward pass of 
-  TAPAS is a bit different in case of a conversational set-up: in that case, you have to feed every training example one by one to the 
-  model, such that the `prev_label_ids` token type ids can be overwritten by the predicted `label_ids` of the model to the previous 
-  question.
-- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective.
-  It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for
-  text generation. Models trained with a causal language modeling (CLM) objective are better in that regard.
+- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell
+  of the table). According to the authors, this usually results in a slightly better performance, and allows you to
+  encode longer sequences without running out of embeddings. If you don't want this, you can set the
+  `reset_position_index_per_cell` parameter of :class:`~transformers.TapasConfig` to False.
+- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a
+  conversational set-up. This means that you can ask follow-up questions such as "what is his age?" related to the
+  previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that
+  case, you have to feed every training example one by one to the model, such that the `prev_label_ids` token type ids
+  can be overwritten by the predicted `label_ids` of the model to the previous question.
+- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore
+  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained
+  with a causal language modeling (CLM) objective are better in that regard.
 
 
 Usage
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-If you just want to perform inference (i.e. making predictions) in a non-conversational setup, you can do the following:
+If you just want to perform inference (i.e. making predictions) in a non-conversational setup, you can do the
+following:
 
 .. code-block::
 

src/transformers/__init__.py

@@ -562,10 +562,10 @@ if is_torch_available():
     )
     from .modeling_tapas import (
         TAPAS_PRETRAINED_MODEL_ARCHIVE_LIST,
-        TapasModel,
         TapasForMaskedLM,
         TapasForQuestionAnswering,
         TapasForSequenceClassification,
+        TapasModel,
         load_tf_weights_in_tapas,
     )
     from .modeling_transfo_xl import (

src/transformers/configuration_tapas.py

@@ -17,20 +17,20 @@
 
 from .configuration_utils import PretrainedConfig
 
+
 TAPAS_PRETRAINED_CONFIG_ARCHIVE_MAP = {"tapas-base": "", "tapas-large": ""}  # to be added  # to be added
 
 
 class TapasConfig(PretrainedConfig):
     r"""
-    This is the configuration class to store the configuration of a :class:`~transformers.TapasModel`.
-    It is used to instantiate a TAPAS model according to the specified arguments, defining the model
-    architecture. Instantiating a configuration with the defaults will yield a similar configuration
-    to that of the TAPAS `tapas-base-finetuned-sqa` architecture. Configuration objects
-    inherit from :class:`~transformers.PreTrainedConfig` and can be used to control the model outputs.
-    Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
+    This is the configuration class to store the configuration of a :class:`~transformers.TapasModel`. It is used to
+    instantiate a TAPAS model according to the specified arguments, defining the model architecture. Instantiating a
+    configuration with the defaults will yield a similar configuration to that of the TAPAS `tapas-base-finetuned-sqa`
+    architecture. Configuration objects inherit from :class:`~transformers.PreTrainedConfig` and can be used to control
+    the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
 
-    Hyperparameters additional to BERT are taken from run_task_main.py and hparam_utils.py of the original implementation.
-    Original implementation available at https://github.com/google-research/tapas/tree/master.
+    Hyperparameters additional to BERT are taken from run_task_main.py and hparam_utils.py of the original
+    implementation. Original implementation available at https://github.com/google-research/tapas/tree/master.
 
     Args:
         vocab_size (:obj:`int`, `optional`, defaults to 30522):
@@ -87,9 +87,9 @@ class TapasConfig(PretrainedConfig):
         average_approximation_function: (:obj:`string`, `optional`, defaults to :obj:`"ratio"`):
             Method to calculate expected average of cells in the relaxed case.
         cell_selection_preference: (:obj:`float`, `optional`, defaults to None):
-            Preference for cell selection in ambiguous cases. Only applicable in case of weak supervision for aggregation (WTQ, WikiSQL).
-            If the total mass of the aggregation probabilities (excluding the "NONE" operator) is higher than this hyperparameter, 
-            then aggregation is predicted for an example.
+            Preference for cell selection in ambiguous cases. Only applicable in case of weak supervision for
+            aggregation (WTQ, WikiSQL). If the total mass of the aggregation probabilities (excluding the "NONE"
+            operator) is higher than this hyperparameter, then aggregation is predicted for an example.
         answer_loss_cutoff: (:obj:`float`, `optional`, defaults to None):
             Ignore examples with answer loss larger than cutoff.
         max_num_rows: (:obj:`int`, `optional`, defaults to 64):

src/transformers/convert_tapas_original_tf_checkpoint_to_pytorch.py

@@ -19,7 +19,12 @@ import argparse
 
 import torch
 
-from transformers import TapasConfig, TapasForQuestionAnswering, TapasForSequenceClassification, load_tf_weights_in_tapas
+from transformers import (
+    TapasConfig,
+    TapasForQuestionAnswering,
+    TapasForSequenceClassification,
+    load_tf_weights_in_tapas,
+)
 from transformers.utils import logging
 
 
@@ -48,7 +53,7 @@ def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, tapas_config_file, pyto
     print("Building PyTorch model from configuration: {}".format(str(config)))
     # model = TapasForMaskedLM(config)
     model = TapasForQuestionAnswering(config)
-    #model = TapasForSequenceClassification(config)
+    # model = TapasForSequenceClassification(config)
 
     # Load weights from tf checkpoint
     load_tf_weights_in_tapas(model, config, tf_checkpoint_path)

src/transformers/modeling_tapas.py

@@ -15,11 +15,11 @@
 """PyTorch TAPAS model. """
 
 
+import enum
 import math
 import os
 from dataclasses import dataclass
 from typing import Optional, Tuple
-import enum
 
 import torch
 import torch.nn as nn
@@ -27,19 +27,15 @@ from torch.nn import CrossEntropyLoss, MSELoss
 
 from .activations import ACT2FN
 from .configuration_tapas import TapasConfig
-from .file_utils import (ModelOutput, 
-                        add_start_docstrings, 
-                        add_start_docstrings_to_model_forward,
-                        replace_return_docstrings,
-                        is_scatter_available,
-                        requires_scatter,
-)
-from .modeling_outputs import (
-    BaseModelOutput, 
-    BaseModelOutputWithPooling, 
-    MaskedLMOutput,
-    SequenceClassifierOutput,
+from .file_utils import (
+    ModelOutput,
+    add_start_docstrings,
+    add_start_docstrings_to_model_forward,
+    is_scatter_available,
+    replace_return_docstrings,
+    requires_scatter,
 )
+from .modeling_outputs import BaseModelOutput, BaseModelOutputWithPooling, MaskedLMOutput, SequenceClassifierOutput
 from .modeling_utils import (
     PreTrainedModel,
     apply_chunking_to_forward,
@@ -48,6 +44,7 @@ from .modeling_utils import (
 )
 from .utils import logging
 
+
 # soft dependency
 if is_scatter_available():
     from torch_scatter import scatter
@@ -75,20 +72,20 @@ class TableQuestionAnsweringOutput(ModelOutput):
 
     Args:
         loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`label_ids` (and possibly :obj:`answer`, :obj:`aggregation_labels`, :obj:`numeric_values` and :obj:`numeric_values_scale` are provided)):
-            Total loss as the sum of the hierarchical cell selection log-likelihood loss and (optionally) the semi-supervised regression loss and (optionally) supervised loss for aggregations.
+            Total loss as the sum of the hierarchical cell selection log-likelihood loss and (optionally) the
+            semi-supervised regression loss and (optionally) supervised loss for aggregations.
         logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`):
             Prediction scores of the cell selection head, for every token.
         logits_aggregation (:obj:`torch.FloatTensor`, `optional`, of shape :obj:`(batch_size, num_aggregation_labels)`):
             Prediction scores of the aggregation head, for every aggregation operator.
         hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``):
             Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
-            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
-            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+            of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of
+            each layer plus the initial embedding outputs.
         attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``):
-            Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
-            :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
-            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
-            heads.
+            Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads,
+            sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the
+            weighted average in the self-attention heads.
     """
 
     loss: Optional[torch.FloatTensor] = None
@@ -99,7 +96,9 @@ class TableQuestionAnsweringOutput(ModelOutput):
 
 
 def load_tf_weights_in_tapas(model, config, tf_checkpoint_path):
-    """Load tf checkpoints in a PyTorch model. This is an adaptation from load_tf_weights_in_bert
+    """
+    Load tf checkpoints in a PyTorch model. This is an adaptation from load_tf_weights_in_bert
+
     - add cell selection and aggregation heads
     - take into account additional token type embedding layers
     """
@@ -148,15 +147,15 @@ def load_tf_weights_in_tapas(model, config, tf_checkpoint_path):
         # since these are not used for classification
         if isinstance(model, TapasForSequenceClassification):
             if any(
-              n
-              in [
-                  "output_bias",
-                  "output_weights",
-              ]
-              for n in name
+                n
+                in [
+                    "output_bias",
+                    "output_weights",
+                ]
+                for n in name
             ):
-              logger.info("Skipping {}".format("/".join(name)))
-              continue
+                logger.info("Skipping {}".format("/".join(name)))
+                continue
         # if first scope name starts with "bert", change it to "tapas"
         if name[0] == "bert":
             name[0] = "tapas"
@@ -233,8 +232,8 @@ def load_tf_weights_in_tapas(model, config, tf_checkpoint_path):
 
 class TapasEmbeddings(nn.Module):
     """
-    Construct the embeddings from word, position and token_type embeddings.
-    Same as BertEmbeddings but with a number of additional token type embeddings to encode tabular structure.
+    Construct the embeddings from word, position and token_type embeddings. Same as BertEmbeddings but with a number of
+    additional token type embeddings to encode tabular structure.
     """
 
     def __init__(self, config):
@@ -632,8 +631,9 @@ class TapasPooler(nn.Module):
 
 
 class TapasPreTrainedModel(PreTrainedModel):
-    """An abstract class to handle weights initialization and
-    a simple interface for downloading and loading pretrained models.
+    """
+    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
+    models.
     """
 
     config_class = TapasConfig
@@ -658,42 +658,40 @@ TAPAS_START_DOCSTRING = r"""
     methods the library implements for all its models (such as downloading or saving, resizing the input embeddings,
     pruning heads etc.)
 
-    This model is also a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`__ subclass.
-    Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general
-    usage and behavior.
+    This model is also a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`__
+    subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to
+    general usage and behavior.
 
     Parameters:
         config (:class:`~transformers.TapasConfig`): Model configuration class with all the parameters of the model.
-            Initializing with a config file does not load the weights associated with the model, only the configuration.
-            Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights.
+            Initializing with a config file does not load the weights associated with the model, only the
+            configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model
+            weights.
 """
 
 TAPAS_INPUTS_DOCSTRING = r"""
     Args:
         input_ids (:obj:`torch.LongTensor` of shape :obj:`({0})`):
-            Indices of input sequence tokens in the vocabulary.
-            Indices can be obtained using :class:`~transformers.TapasTokenizer`.
-            See :meth:`transformers.PreTrainedTokenizer.encode` and
-            :meth:`transformers.PreTrainedTokenizer.__call__` for details.
-            `What are input IDs? <../glossary.html#input-ids>`__
+            Indices of input sequence tokens in the vocabulary. Indices can be obtained using
+            :class:`~transformers.TapasTokenizer`. See :meth:`transformers.PreTrainedTokenizer.encode` and
+            :meth:`transformers.PreTrainedTokenizer.__call__` for details. `What are input IDs?
+            <../glossary.html#input-ids>`__
         attention_mask (:obj:`torch.FloatTensor` of shape :obj:`({0})`, `optional`):
-            Mask to avoid performing attention on padding token indices.
-            Mask values selected in ``[0, 1]``:
-            - 1 for tokens that are **not masked**,
-            - 0 for tokens that are **masked**.
-            `What are attention masks? <../glossary.html#attention-mask>`__
+            Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: - 1 for
+            tokens that are **not masked**, - 0 for tokens that are **masked**. `What are attention masks?
+            <../glossary.html#attention-mask>`__
         token_type_ids (:obj:`torch.LongTensor` of shape :obj:`({0}, 7)`, `optional`):
-            Token indices that encode tabular structure. Indices can be obtained using :class:`~transformers.TapasTokenizer`. See this class for more info. 
-            `What are token type IDs? <../glossary.html#token-type-ids>`_
+            Token indices that encode tabular structure. Indices can be obtained using
+            :class:`~transformers.TapasTokenizer`. See this class for more info. `What are token type IDs?
+            <../glossary.html#token-type-ids>`_
         position_ids (:obj:`torch.LongTensor` of shape :obj:`({0})`, `optional`):
-            Indices of positions of each input sequence tokens in the position embeddings. If ``reset_position_index_per_cell`` of :class:`~transformers.TapasConfig` is set to ``True``, relative position embeddings will be used.
-            Selected in the range ``[0, config.max_position_embeddings - 1]``.
-            `What are position IDs? <../glossary.html#position-ids>`_
+            Indices of positions of each input sequence tokens in the position embeddings. If
+            ``reset_position_index_per_cell`` of :class:`~transformers.TapasConfig` is set to ``True``, relative
+            position embeddings will be used. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What
+            are position IDs? <../glossary.html#position-ids>`_
         head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`):
-            Mask to nullify selected heads of the self-attention modules.
-            Mask values selected in ``[0, 1]``:
-            - 1 indicates the head is **not masked**,
-            - 0 indicates the head is **masked**.
+            Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: - 1
+            indicates the head is **not masked**, - 0 indicates the head is **masked**.
         inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`({0}, hidden_size)`, `optional`):
             Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation.
             This is useful if you want more control over how to convert :obj:`input_ids` indices into associated
@@ -715,19 +713,18 @@ TAPAS_INPUTS_DOCSTRING = r"""
 )
 class TapasModel(TapasPreTrainedModel):
     """
-    This class is a small change compared to :class:`~transformers.BertModel`, taking into account the additional token type ids.
+    This class is a small change compared to :class:`~transformers.BertModel`, taking into account the additional token
+    type ids.
 
-    The model can behave as an encoder (with only self-attention) as well
-    as a decoder, in which case a layer of cross-attention is added between
-    the self-attention layers, following the architecture described in `Attention is all you need
-    <https://arxiv.org/abs/1706.03762>`__ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
-    Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.
+    The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of
+    cross-attention is added between the self-attention layers, following the architecture described in `Attention is
+    all you need <https://arxiv.org/abs/1706.03762>`__ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit,
+    Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.
 
-    To behave as an decoder the model needs to be initialized with the
-    :obj:`is_decoder` argument of the configuration set to :obj:`True`.
-    To be used in a Seq2Seq model, the model needs to initialized with both :obj:`is_decoder`
-    argument and :obj:`add_cross_attention` set to :obj:`True`; an
-    :obj:`encoder_hidden_states` is then expected as an input to the forward pass.
+    To behave as an decoder the model needs to be initialized with the :obj:`is_decoder` argument of the configuration
+    set to :obj:`True`. To be used in a Seq2Seq model, the model needs to initialized with both :obj:`is_decoder`
+    argument and :obj:`add_cross_attention` set to :obj:`True`; an :obj:`encoder_hidden_states` is then expected as an
+    input to the forward pass.
     """
 
     config_class = TapasConfig
@@ -751,9 +748,9 @@ class TapasModel(TapasPreTrainedModel):
         self.embeddings.word_embeddings = value
 
     def _prune_heads(self, heads_to_prune):
-        """Prunes heads of the model.
-        heads_to_prune: dict of {layer_num: list of heads to prune in this layer}
-        See base class PreTrainedModel
+        """
+        Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base
+        class PreTrainedModel
         """
         for layer, heads in heads_to_prune.items():
             self.encoder.layer[layer].attention.prune_heads(heads)
@@ -903,10 +900,9 @@ class TapasForMaskedLM(TapasPreTrainedModel):
     ):
         r"""
         labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`):
-            Labels for computing the masked language modeling loss.
-            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
-            Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
-            in ``[0, ..., config.vocab_size]``
+            Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ...,
+            config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored
+            (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]``
 
         Returns:
 
@@ -989,8 +985,11 @@ class TapasLMHead(nn.Module):
 
 
 @add_start_docstrings(
-    """Tapas Model with a cell selection head and optionally aggregation head on top for question-answering 
-    tasks on tables (linear layers on top of the hidden-states output to compute `logits` and optionally `logits_aggregation`), e.g. for SQA, WTQ or WikiSQL tasks. """,
+    """
+    Tapas Model with a cell selection head and optionally aggregation head on top for question-answering tasks on
+    tables (linear layers on top of the hidden-states output to compute `logits` and optionally `logits_aggregation`),
+    e.g. for SQA, WTQ or WikiSQL tasks.
+    """,
     TAPAS_START_DOCSTRING,
 )
 class TapasForQuestionAnswering(TapasPreTrainedModel):
@@ -1049,21 +1048,23 @@ class TapasForQuestionAnswering(TapasPreTrainedModel):
     ):
         r"""
         table_mask (:obj:`torch.LongTensor` of shape :obj:`(batch_size, seq_length)`, `optional`):
-            Mask for the table. Indicates which tokens belong to the table (1). Question tokens, table headers and padding are 0.
+            Mask for the table. Indicates which tokens belong to the table (1). Question tokens, table headers and
+            padding are 0.
         label_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, seq_length)`, `optional`):
-            Labels per token for computing the hierarchical cell selection loss. This encodes the positions of the answer appearing in the table. Can be obtained using :class:`~transformers.TapasTokenizer`.
-            - 1 for tokens that are **part of the answer**,
-            - 0 for tokens that are **not part of the answer**.
+            Labels per token for computing the hierarchical cell selection loss. This encodes the positions of the
+            answer appearing in the table. Can be obtained using :class:`~transformers.TapasTokenizer`. - 1 for tokens
+            that are **part of the answer**, - 0 for tokens that are **not part of the answer**.
         aggregation_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, )`, `optional`):
-            Aggregation function index for every example in the batch for computing the aggregation loss.
-            Indices should be in :obj:`[0, ..., config.num_aggregation_labels - 1]`.
-            Only required in case of strong supervision for aggregation (WikiSQL-SUPERVISED). 
+            Aggregation function index for every example in the batch for computing the aggregation loss. Indices
+            should be in :obj:`[0, ..., config.num_aggregation_labels - 1]`. Only required in case of strong
+            supervision for aggregation (WikiSQL-SUPERVISED).
         answer (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, )`, `optional`):
-            Answer for every example in the batch. NaN if there is no scalar answer. 
-            Only required in case of weak supervision (WTQ, WikiSQL) to calculate the aggregate mask and regression loss. 
+            Answer for every example in the batch. NaN if there is no scalar answer. Only required in case of weak
+            supervision (WTQ, WikiSQL) to calculate the aggregate mask and regression loss.
         numeric_values (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, seq_length)`, `optional`):
-            Numeric values of every token, NaN for tokens which are not numeric values. Can be obtained using :class:`~transformers.TapasTokenizer`. 
-            Only required in case of weak supervision for aggregation (WTQ, WikiSQL) to calculate the regression loss.
+            Numeric values of every token, NaN for tokens which are not numeric values. Can be obtained using
+            :class:`~transformers.TapasTokenizer`. Only required in case of weak supervision for aggregation (WTQ,
+            WikiSQL) to calculate the regression loss.
         numeric_values_scale (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, seq_length)`, `optional`):
             Scale of the numeric values of every token. Can be obtained using :class:`~transformers.TapasTokenizer`.
             Only required in case of weak supervision for aggregation (WTQ, WikiSQL) to calculate the regression loss.
@@ -1160,9 +1161,7 @@ class TapasForQuestionAnswering(TapasPreTrainedModel):
         cell_mask, _ = reduce_mean(input_mask_float, cell_index)
 
         # Compute logits per token. These are used to select individual cells.
-        logits = compute_token_logits(
-            sequence_output, self.config.temperature, self.output_weights, self.output_bias
-        )
+        logits = compute_token_logits(sequence_output, self.config.temperature, self.output_weights, self.output_bias)
 
         # Compute logits per column. These are used to select a column.
         column_logits = None
@@ -1208,7 +1207,7 @@ class TapasForQuestionAnswering(TapasPreTrainedModel):
                         pooled_output,
                         self.config.cell_selection_preference,
                         label_ids,
-                        self.aggregation_classifier
+                        self.aggregation_classifier,
                     )
                 else:
                     raise ValueError("You have to specify answers in order to calculate the aggregate mask")
@@ -1262,13 +1261,15 @@ class TapasForQuestionAnswering(TapasPreTrainedModel):
                             logits_aggregation, aggregate_mask, aggregation_labels, self.config
                         )
                     else:
-                        raise ValueError("You have to specify aggregation labels in order to calculate the aggregation loss")
+                        raise ValueError(
+                            "You have to specify aggregation labels in order to calculate the aggregation loss"
+                        )
                 else:
                     # Set aggregation labels to zeros
                     aggregation_labels = torch.zeros(label_ids.shape[0], dtype=torch.long, device=label_ids.device)
                     per_example_additional_loss = _calculate_aggregation_loss(
-                            logits_aggregation, aggregate_mask, aggregation_labels, self.config
-                        )
+                        logits_aggregation, aggregate_mask, aggregation_labels, self.config
+                    )
 
                 if self.config.use_answer_as_supervision:
                     if numeric_values is not None and numeric_values_scale is not None:
@@ -1288,7 +1289,9 @@ class TapasForQuestionAnswering(TapasPreTrainedModel):
                         # Zero loss for examples with answer_loss > cutoff.
                         per_example_additional_loss *= large_answer_loss_mask
                     else:
-                        raise ValueError("You have to specify numeric values and numeric values scale in order to calculate the regression loss")
+                        raise ValueError(
+                            "You have to specify numeric values and numeric values scale in order to calculate the regression loss"
+                        )
 
                 total_loss += torch.mean(per_example_additional_loss)
 
@@ -1310,9 +1313,12 @@ class TapasForQuestionAnswering(TapasPreTrainedModel):
             attentions=outputs.attentions,
         )
 
+
 @add_start_docstrings(
-    """Tapas Model with a sequence classification head on top (a linear layer on top of
-    the pooled output), e.g. for TabFact (Chen et al., 2020). """,
+    """
+    Tapas Model with a sequence classification head on top (a linear layer on top of the pooled output), e.g. for
+    TabFact (Chen et al., 2020).
+    """,
     TAPAS_START_DOCSTRING,
 )
 class TapasForSequenceClassification(TapasPreTrainedModel):
@@ -1343,11 +1349,10 @@ class TapasForSequenceClassification(TapasPreTrainedModel):
     ):
         r"""
         labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`):
-            Labels for computing the sequence classification/regression loss.
-            Indices should be in :obj:`[0, ..., config.num_labels - 1]`.
-            If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss),
-            If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
-            Note: this is called "classification_class_index" in the original implementation. 
+            Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[0, ...,
+            config.num_labels - 1]`. If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss),
+            If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy). Note: this is called
+            "classification_class_index" in the original implementation.
 
         Returns:
 
@@ -1411,8 +1416,10 @@ class TapasForSequenceClassification(TapasPreTrainedModel):
             attentions=outputs.attentions,
         )
 
+
 """ TAPAS utilities."""
 
+
 class AverageApproximationFunction(str, enum.Enum):
     RATIO = "ratio"
     FIRST_ORDER = "first_order"
@@ -1426,17 +1433,19 @@ class IndexMap(object):
     """Index grouping entries within a tensor."""
 
     def __init__(self, indices, num_segments, batch_dims=0):
-        """Creates an index.
+        """
+        Creates an index
+
         Args:
             indices (:obj:`torch.LongTensor`, same shape as `values`):
                 Tensor containing the indices.
             num_segments (:obj:`torch.LongTensor`):
-                Scalar tensor, the number of segments. All elements in a batched segmented tensor
-                must have the same number of segments (although many segments can be empty).
+                Scalar tensor, the number of segments. All elements in a batched segmented tensor must have the same
+                number of segments (although many segments can be empty).
             batch_dims (:obj:`int`, `optional`, defaults to 0):
-                The number of batch dimensions. The first `batch_dims` dimensions of a SegmentedTensor
-                are treated as batch dimensions. Segments in different batch elements are always distinct
-                even if they have the same index.
+                The number of batch dimensions. The first `batch_dims` dimensions of a SegmentedTensor are treated as
+                batch dimensions. Segments in different batch elements are always distinct even if they have the same
+                index.
         """
         self.indices = torch.as_tensor(indices)
         self.num_segments = torch.as_tensor(num_segments, device=indices.device)
@@ -1450,14 +1459,13 @@ class ProductIndexMap(IndexMap):
     """The product of two indices."""
 
     def __init__(self, outer_index, inner_index):
-        """Combines indices i and j into pairs (i, j).
-        The result is an index where each segment (i, j) is the intersection of
-        segments i and j. For example if the inputs represent table cells indexed by
-        respectively rows and columns the output will be a table indexed by
-        (row, column) pairs, i.e. by cell.
-        The implementation combines indices {0, .., n - 1} and {0, .., m - 1} into
-        {0, .., nm - 1}. The output has `num_segments` equal to
-            `outer_index.num_segments` * `inner_index.num_segments`.
+        """
+        Combines indices i and j into pairs (i, j). The result is an index where each segment (i, j) is the
+        intersection of segments i and j. For example if the inputs represent table cells indexed by respectively rows
+        and columns the output will be a table indexed by (row, column) pairs, i.e. by cell. The implementation
+        combines indices {0, .., n - 1} and {0, .., m - 1} into {0, .., nm - 1}. The output has `num_segments` equal to
+        `outer_index.num_segments` * `inner_index.num_segments`
+
         Args:
             outer_index (:obj:`IndexMap`):
                 IndexMap.
@@ -1496,17 +1504,18 @@ class ProductIndexMap(IndexMap):
 
 
 def gather(values, index, name="segmented_gather"):
-    """Gathers from `values` using the index map.
-    For each element in the domain of the index map this operation looks up a
-    value for that index in `values`. Two elements from the same segment always
-    get assigned the same value.
+    """
+    Gathers from `values` using the index map. For each element in the domain of the index map this operation looks up
+    a value for that index in `values`. Two elements from the same segment always get assigned the same value
+
     Args:
         values (:obj:`torch.Tensor` of shape (B1, ..., Bn, num_segments, V1, ...)):
             Tensor with segment values.
         index (:obj:`IndexMap` of shape (B1, ..., Bn, I1, ..., Ik)):
             IndexMap.
         name (:obj:`str`, `optional`, defaults to 'segmented_gather'):
-            Name for the operation. Currently not used.
+            Name for the operation. Currently not used
+
     Returns:
         :obj:`tuple(torch.Tensor)`: Tensor of shape (B1, ..., Bn, I1, ..., Ik, V1, ...) with the gathered values.
     """
@@ -1528,16 +1537,18 @@ def gather(values, index, name="segmented_gather"):
 
 
 def flatten(index, name="segmented_flatten"):
-    """Flattens a batched index map (which is typically of shape batch_size, seq_length) to a 1d index map.
-    This operation relabels the segments to keep batch elements distinct. The k-th
-    batch element will have indices shifted by `num_segments` * (k - 1). The
-    result is a tensor with `num_segments` multiplied by the number of elements
-    in the batch.
+    """
+    Flattens a batched index map (which is typically of shape batch_size, seq_length) to a 1d index map. This operation
+    relabels the segments to keep batch elements distinct. The k-th batch element will have indices shifted by
+    `num_segments` * (k - 1). The result is a tensor with `num_segments` multiplied by the number of elements in the
+    batch
+
     Args:
         index (:obj:`IndexMap`):
             IndexMap to flatten.
         name (:obj:`str`, `optional`, defaults to 'segmented_flatten'):
-            Name for the operation. Currently not used.
+            Name for the operation. Currently not used
+
     Returns:
         (:obj:`IndexMap`): The flattened IndexMap.
     """
@@ -1555,14 +1566,17 @@ def flatten(index, name="segmented_flatten"):
 
 
 def range_index_map(batch_shape, num_segments, name="range_index_map"):
-    """Constructs an index map equal to range(num_segments).
+    """
+    Constructs an index map equal to range(num_segments)
+
     Args:
         batch_shape (:obj:`torch.Size`):
             Batch shape
         num_segments (:obj:`int`):
             Number of segments
         name (:obj:`str`, `optional`, defaults to 'range_index_map'):
-            Name for the operation. Currently not used.
+            Name for the operation. Currently not used
+
     Returns:
         (:obj:`IndexMap`): IndexMap of shape batch_shape with elements equal to range(num_segments).
     """
@@ -1593,7 +1607,9 @@ def range_index_map(batch_shape, num_segments, name="range_index_map"):
 
 
 def _segment_reduce(values, index, segment_reduce_fn, name):
-    """Applies a segment reduction segment-wise.
+    """
+    Applies a segment reduction segment-wise
+
     Args:
         values (:obj:`torch.Tensor`):
             Tensor with segment values.
@@ -1602,7 +1618,8 @@ def _segment_reduce(values, index, segment_reduce_fn, name):
         segment_reduce_fn (:obj:`str`):
             Name for the reduce operation. One of "sum", "mean", "max" or "min".
         name (:obj:`str`):
-            Name for the operation. Currently not used.
+            Name for the operation. Currently not used
+
     Returns:
         (:obj:`IndexMap`): IndexMap of shape batch_shape with elements equal to range(num_segments).
     """
@@ -1641,99 +1658,99 @@ def _segment_reduce(values, index, segment_reduce_fn, name):
 
 
 def reduce_sum(values, index, name="segmented_reduce_sum"):
-    """Sums a tensor over its segments.
-    Outputs 0 for empty segments.
-    This operations computes the sum over segments, with support for:
+    """
+    Sums a tensor over its segments. Outputs 0 for empty segments. This operations computes the sum over segments, with
+    support for:
+
         - Batching using the first dimensions [B1, B2, ..., Bn]. Each element in
-        a batch can have different indices.
-        - Vectorization using the last dimension [V1, V2, ...]. If they are present
-        the output will be a sum of vectors rather than scalars.
-    Only the middle dimensions [I1, ..., Ik] are reduced by the operation.
+        a batch can have different indices. - Vectorization using the last dimension [V1, V2, ...]. If they are present
+        the output will be a sum of vectors rather than scalars. Only the middle dimensions [I1, ..., Ik] are reduced
+        by the operation
+
     Args:
         values (:obj:`torch.Tensor` of shape [B1, B2, ..., Bn, I1, .., Ik, V1, V2, ..]):
             Tensor containing the values of which the sum must be taken segment-wise.
         index (:obj:`IndexMap`, indices are of shape [B1, B2, ..., Bn, I1, .., Ik].):
             Index defining the segments.
         name (:obj:`str`, `optional`, defaults to 'segmented_reduce_sum'):
-            Name for the operation. Currently not used.
+            Name for the operation. Currently not used
+
     Returns:
-        output_values (:obj:`torch.Tensor`of shape [B1, B2, ..., Bn, num_segments, V1, V2, ..]):
-            Tensor containing the output values.
-        output_index (:obj:`IndexMap`):
-            IndexMap with shape [B1, B2, ..., Bn, num_segments].       .
+        output_values (:obj:`torch.Tensor`of shape [B1, B2, ..., Bn, num_segments, V1, V2, ..]): Tensor containing the
+        output values. output_index (:obj:`IndexMap`): IndexMap with shape [B1, B2, ..., Bn, num_segments]. .
     """
     return _segment_reduce(values, index, "sum", name)
 
 
 def reduce_mean(values, index, name="segmented_reduce_mean"):
-    """Averages a tensor over its segments.
-    Outputs 0 for empty segments.
-    This operations computes the mean over segments, with support for:
+    """
+    Averages a tensor over its segments. Outputs 0 for empty segments. This operations computes the mean over segments,
+    with support for:
+
         - Batching using the first dimensions [B1, B2, ..., Bn]. Each element in
-        a batch can have different indices.
-        - Vectorization using the last dimension [V1, V2, ...]. If they are present
-        the output will be a mean of vectors rather than scalars.
-    Only the middle dimensions [I1, ..., Ik] are reduced by the operation.
+        a batch can have different indices. - Vectorization using the last dimension [V1, V2, ...]. If they are present
+        the output will be a mean of vectors rather than scalars. Only the middle dimensions [I1, ..., Ik] are reduced
+        by the operation
+
     Args:
         values (:obj:`torch.Tensor` of shape [B1, B2, ..., Bn, I1, .., Ik, V1, V2, ..]):
             Tensor containing the values of which the mean must be taken segment-wise.
         index (:obj:`IndexMap`, indices are of shape [B1, B2, ..., Bn, I1, .., Ik].):
             Index defining the segments.
         name (:obj:`str`, `optional`, defaults to 'segmented_reduce_sum'):
-            Name for the operation. Currently not used.
+            Name for the operation. Currently not used
+
     Returns:
-        output_values (:obj:`torch.Tensor`of shape [B1, B2, ..., Bn, num_segments, V1, V2, ..]):
-            Tensor containing the output values.
-        output_index (:obj:`IndexMap`):
-            IndexMap with shape [B1, B2, ..., Bn, num_segments].
+        output_values (:obj:`torch.Tensor`of shape [B1, B2, ..., Bn, num_segments, V1, V2, ..]): Tensor containing the
+        output values. output_index (:obj:`IndexMap`): IndexMap with shape [B1, B2, ..., Bn, num_segments].
     """
     return _segment_reduce(values, index, "mean", name)
 
 
 def reduce_max(values, index, name="segmented_reduce_max"):
-    """Computes the maximum over segments.
-    This operations computes the maximum over segments, with support for:
+    """
+    Computes the maximum over segments. This operations computes the maximum over segments, with support for:
+
         - Batching using the first dimensions [B1, B2, ..., Bn]. Each element in
-        a batch can have different indices.
-        - Vectorization using the last dimension [V1, V2, ...]. If they are present
-        the output will be an element-wise maximum of vectors rather than scalars.
-    Only the middle dimensions [I1, ..., Ik] are reduced by the operation.
+        a batch can have different indices. - Vectorization using the last dimension [V1, V2, ...]. If they are present
+        the output will be an element-wise maximum of vectors rather than scalars. Only the middle dimensions [I1, ...,
+        Ik] are reduced by the operation
+
     Args:
         values (:obj:`torch.Tensor` of shape [B1, B2, ..., Bn, I1, .., Ik, V1, V2, ..]):
             Tensor containing the values of which the max must be taken segment-wise.
         index (:obj:`IndexMap`, indices are of shape [B1, B2, ..., Bn, I1, .., Ik].):
             Index defining the segments.
         name (:obj:`str`, `optional`, defaults to 'segmented_reduce_sum'):
-            Name for the operation. Currently not used.
+            Name for the operation. Currently not used
+
     Returns:
-        output_values (:obj:`torch.Tensor`of shape [B1, B2, ..., Bn, num_segments, V1, V2, ..]):
-            Tensor containing the output values.
-        output_index (:obj:`IndexMap`):
-            IndexMap with shape [B1, B2, ..., Bn, num_segments].
+        output_values (:obj:`torch.Tensor`of shape [B1, B2, ..., Bn, num_segments, V1, V2, ..]): Tensor containing the
+        output values. output_index (:obj:`IndexMap`): IndexMap with shape [B1, B2, ..., Bn, num_segments].
     """
     return _segment_reduce(values, index, "max", name)
 
 
 def reduce_min(values, index, name="segmented_reduce_min"):
-    """Computes the minimum over segments.
-    This operations computes the maximum over segments, with support for:
+    """
+    Computes the minimum over segments. This operations computes the maximum over segments, with support for:
+
         - Batching using the first dimensions [B1, B2, ..., Bn]. Each element in
-        a batch can have different indices.
-        - Vectorization using the last dimension [V1, V2, ...]. If they are present
-        the output will be an element-wise maximum of vectors rather than scalars.
-    Only the middle dimensions [I1, ..., Ik] are reduced by the operation.
+        a batch can have different indices. - Vectorization using the last dimension [V1, V2, ...]. If they are present
+        the output will be an element-wise maximum of vectors rather than scalars. Only the middle dimensions [I1, ...,
+        Ik] are reduced by the operation
+
     Args:
         values (:obj:`torch.Tensor` of shape [B1, B2, ..., Bn, I1, .., Ik, V1, V2, ..]):
             Tensor containing the values of which the min must be taken segment-wise.
         index (:obj:`IndexMap`, indices are of shape [B1, B2, ..., Bn, I1, .., Ik].):
             Index defining the segments.
         name (:obj:`str`, `optional`, defaults to 'segmented_reduce_sum'):
-            Name for the operation. Currently not used.
+            Name for the operation. Currently not used
+
     Returns:
-        output_values (:obj:`torch.Tensor`of shape [B1, B2, ..., Bn, num_segments, V1, V2, ..]):
-            Tensor containing the output values.
-        output_index (:obj:`IndexMap`):
-            IndexMap with shape [B1, B2, ..., Bn, num_segments].
+        output_values (:obj:`torch.Tensor`of shape [B1, B2, ..., Bn, num_segments, V1, V2, ..]): Tensor containing the
+        output values. output_index (:obj:`IndexMap`): IndexMap with shape [B1, B2, ..., Bn, num_segments].
     """
     return _segment_reduce(values, index, "min", name)
 
@@ -1744,7 +1761,8 @@ def reduce_min(values, index, name="segmented_reduce_min"):
 def compute_column_logits(
     sequence_output, column_output_weights, column_output_bias, cell_index, cell_mask, allow_empty_column_selection
 ):
-    """Computes the column logits.
+    """
+    Computes the column logits.
 
     Args:
         sequence_output (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
@@ -1758,10 +1776,11 @@ def compute_column_logits(
         cell_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, max_num_rows * max_num_cols)`):
             Mask for cells that exist in the table (i.e. that are not padding).
         allow_empty_column_selection (:obj:`bool`):
-            Whether to allow not to select any column.
+            Whether to allow not to select any column
+
     Returns:
-        column_logits (:obj:`torch.FloatTensor`of shape :obj:`(batch_size, max_num_cols)`):
-            Tensor containing the column logits for every example in the batch.
+        column_logits (:obj:`torch.FloatTensor`of shape :obj:`(batch_size, max_num_cols)`): Tensor containing the
+        column logits for every example in the batch.
     """
 
     # First, compute the token logits (batch_size, seq_len) - without temperature
@@ -1792,10 +1811,10 @@ def compute_column_logits(
 
 
 def _single_column_cell_selection_loss(token_logits, column_logits, label_ids, cell_index, col_index, cell_mask):
-    """Computes the loss for cell selection constrained to a single column.
-    The loss is a hierarchical log-likelihood. The model first predicts a column
-    and then selects cells within that column (conditioned on the column). Cells
-    outside the selected column are never selected.
+    """
+    Computes the loss for cell selection constrained to a single column. The loss is a hierarchical log-likelihood. The
+    model first predicts a column and then selects cells within that column (conditioned on the column). Cells outside
+    the selected column are never selected.
 
     Args:
         token_logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`):
@@ -1812,11 +1831,10 @@ def _single_column_cell_selection_loss(token_logits, column_logits, label_ids, c
             Mask for cells that exist in the table (i.e. that are not padding).
 
     Returns:
-        selection_loss_per_example (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`):
-            Loss for each example.
-        logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`):
-            New logits which are only allowed to select cells in a single column. Logits outside of the most likely
-            column according to `column_logits` will be set to a very low value (such that the probabilities are 0).
+        selection_loss_per_example (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`): Loss for each example.
+        logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`): New logits which are only
+        allowed to select cells in a single column. Logits outside of the most likely column according to
+        `column_logits` will be set to a very low value (such that the probabilities are 0).
     """
     ## Part 1: column loss
 
@@ -1903,7 +1921,9 @@ def _single_column_cell_selection_loss(token_logits, column_logits, label_ids, c
 
 
 def compute_token_logits(sequence_output, temperature, output_weights, output_bias):
-    """Computes logits per token.
+    """
+    Computes logits per token
+
     Args:
         sequence_output (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
             Also known as last_hidden_state. Sequence of hidden-states at the output of the last layer of the model.
@@ -1912,10 +1932,10 @@ def compute_token_logits(sequence_output, temperature, output_weights, output_bi
         output_weights (:obj:`torch.FloatTensor` of shape :obj:`(hidden_size,)`):
             Weights of the linear layer for cell selection.
         output_bias (:obj:`torch.FloatTensor` of shape :obj:`()`):
-            Bias of the linear layer for cell selection.
+            Bias of the linear layer for cell selection
+
     Returns:
-        logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`):
-            Logits per token.
+        logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`): Logits per token.
     """
     logits = (torch.einsum("bsj,j->bs", sequence_output, output_weights) + output_bias) / temperature
 
@@ -1923,17 +1943,16 @@ def compute_token_logits(sequence_output, temperature, output_weights, output_bi
 
 
 def _calculate_aggregate_mask(answer, pooled_output, cell_selection_preference, label_ids, aggregation_classifier):
-    """Finds examples where the model should select cells with no aggregation.
+    """
+    Finds examples where the model should select cells with no aggregation.
+
+    Returns a mask that determines for which examples should the model select answers directly from the table, without
+    any aggregation function. If the answer is a piece of text the case is unambiguous as aggregation functions only
+    apply to numbers. If the answer is a number but does not appear in the table then we must use some aggregation
+    case. The ambiguous case is when the answer is a number that also appears in the table. In this case we use the
+    aggregation function probabilities predicted by the model to decide whether to select or aggregate. The threshold
+    for this is a hyperparameter `cell_selection_preference
 
-    Returns a mask that determines for which examples should the model select
-    answers directly from the table, without any aggregation function. If the
-    answer is a piece of text the case is unambiguous as aggregation functions
-    only apply to numbers. If the answer is a number but does not appear in the
-    table then we must use some aggregation case. The ambiguous case is when the
-    answer is a number that also appears in the table. In this case we use the
-    aggregation function probabilities predicted by the model to decide whether
-    to select or aggregate. The threshold for this is a hyperparameter
-    `cell_selection_preference`
     Args:
         answer (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, )`):
             Answer for every example in the batch. Nan if there is no scalar answer.
@@ -1942,12 +1961,11 @@ def _calculate_aggregate_mask(answer, pooled_output, cell_selection_preference,
         cell_selection_preference (:obj:`float`):
             Preference for cell selection in ambiguous cases.
         label_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`):
-            Labels per token.
-       aggregation_classifier (:obj:`torch.nn.Linear`):
-            Aggregation head. 
+            Labels per token. aggregation_classifier (:obj:`torch.nn.Linear`): Aggregation head
+
     Returns:
-        aggregate_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`):
-            A mask set to 1 for examples that should use aggregation functions.
+        aggregate_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`): A mask set to 1 for examples that
+        should use aggregation functions.
     """
     # torch.FloatTensor(batch_size,)
     aggregate_mask_init = torch.logical_not(torch.isnan(answer)).type(torch.FloatTensor).to(answer.device)
@@ -1976,13 +1994,13 @@ def _calculate_aggregate_mask(answer, pooled_output, cell_selection_preference,
 
 
 def _calculate_aggregation_loss_known(logits_aggregation, aggregate_mask, aggregation_function_id, config):
-    """Calculates aggregation loss when its type is known during training.
+    """
+    Calculates aggregation loss when its type is known during training.
+
+    In the weakly supervised setting, the only known information is that for cell selection examples, "no aggregation"
+    should be predicted. For other examples (those that require aggregation), no loss is accumulated. In the setting
+    where aggregation type is always known, standard cross entropy loss is accumulated for all examples
 
-    In the weakly supervised setting, the only known information is that for
-    cell selection examples, "no aggregation" should be predicted. For other
-    examples (those that require aggregation), no loss is accumulated.
-    In the setting where aggregation type is always known, standard cross entropy
-    loss is accumulated for all examples.
     Args:
         logits_aggregation (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_aggregation_labels)`):
             Logits per aggregation operation.
@@ -1991,10 +2009,11 @@ def _calculate_aggregation_loss_known(logits_aggregation, aggregate_mask, aggreg
         aggregation_function_id (:obj:`torch.LongTensor` of shape :obj:`(batch_size, )`):
             Aggregation function id for every example in the batch.
         config (:class:`~transformers.TapasConfig`):
-            Model configuration class with all the parameters of the model.
+            Model configuration class with all the parameters of the model
+
     Returns:
-        aggregation_loss_known (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`):
-            Aggregation loss (when its type is known during training) per example.
+        aggregation_loss_known (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`): Aggregation loss (when its
+        type is known during training) per example.
     """
     if config.use_answer_as_supervision:
         # Prepare "no aggregation" targets for cell selection examples.
@@ -2019,15 +2038,18 @@ def _calculate_aggregation_loss_known(logits_aggregation, aggregate_mask, aggreg
 
 
 def _calculate_aggregation_loss_unknown(logits_aggregation, aggregate_mask):
-    """Calculates aggregation loss in the case of answer supervision.
+    """
+    Calculates aggregation loss in the case of answer supervision
+
     Args:
         logits_aggregation (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_aggregation_labels)`):
             Logits per aggregation operation.
         aggregate_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, )`):
-            A mask set to 1 for examples that should use aggregation functions.
+            A mask set to 1 for examples that should use aggregation functions
+
     Returns:
-        aggregation_loss_unknown (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`):
-            Aggregation loss (in case of answer supervision) per example.
+        aggregation_loss_unknown (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`): Aggregation loss (in case of
+        answer supervision) per example.
     """
 
     dist_aggregation = torch.distributions.categorical.Categorical(logits=logits_aggregation)
@@ -2041,7 +2063,9 @@ def _calculate_aggregation_loss_unknown(logits_aggregation, aggregate_mask):
 
 
 def _calculate_aggregation_loss(logits_aggregation, aggregate_mask, aggregation_function_id, config):
-    """Calculates the aggregation loss per example.
+    """
+    Calculates the aggregation loss per example
+
     Args:
         logits_aggregation (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_aggregation_labels)`):
             Logits per aggregation operation.
@@ -2050,10 +2074,10 @@ def _calculate_aggregation_loss(logits_aggregation, aggregate_mask, aggregation_
         aggregation_function_id (:obj:`torch.LongTensor` of shape :obj:`(batch_size, )`):
             Aggregation function id for every example in the batch.
         config (:class:`~transformers.TapasConfig`):
-            Model configuration class with all the parameters of the model.
+            Model configuration class with all the parameters of the model
+
     Returns:
-        aggregation_loss (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`):
-            Aggregation loss per example.
+        aggregation_loss (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`): Aggregation loss per example.
     """
     per_example_aggregation_loss = _calculate_aggregation_loss_known(
         logits_aggregation, aggregate_mask, aggregation_function_id, config
@@ -2068,7 +2092,9 @@ def _calculate_aggregation_loss(logits_aggregation, aggregate_mask, aggregation_
 def _calculate_expected_result(
     dist_per_cell, numeric_values, numeric_values_scale, input_mask_float, logits_aggregation, config
 ):
-    """Calculate the expected result given cell and aggregation probabilities.
+    """
+    Calculate the expected result given cell and aggregation probabilities
+
     Args:
         dist_per_cell (:obj:`torch.distributions.Bernoulli`):
             Cell selection distribution for each cell.
@@ -2081,10 +2107,10 @@ def _calculate_expected_result(
         logits_aggregation (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_aggregation_labels)`):
             Logits per aggregation operation.
         config (:class:`~transformers.TapasConfig`):
-            Model configuration class with all the parameters of the model.
+            Model configuration class with all the parameters of the model
+
     Returns:
-        expected_result (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`):
-            The expected result per example.
+        expected_result (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`): The expected result per example.
     """
     if config.use_gumbel_for_cells:
         gumbel_dist = torch.distributions.RelaxedBernoulli(
@@ -2163,7 +2189,9 @@ def _calculate_regression_loss(
     logits_aggregation,
     config,
 ):
-    """Calculates the regression loss per example.
+    """
+    Calculates the regression loss per example
+
     Args:
         answer (:obj: `torch.FloatTensor` of shape :obj:`(batch_size,)`):
             Answer for every example in the batch. Nan if there is no scalar answer.
@@ -2180,12 +2208,12 @@ def _calculate_regression_loss(
         logits_aggregation (:obj: `torch.FloatTensor` of shape :obj:`(batch_size, num_aggregation_labels)`):
             Logits per aggregation operation.
         config (:class:`~transformers.TapasConfig`):
-            Model configuration class with all the parameters of the model.
+            Model configuration class with all the parameters of the model
+
     Returns:
-        per_example_answer_loss_scaled (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`):
-            Scales answer loss for each example in the batch.
-        large_answer_loss_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`):
-            A mask which is 1 for examples for which their answer loss is larger than the answer_loss_cutoff.
+        per_example_answer_loss_scaled (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`): Scales answer loss for
+        each example in the batch. large_answer_loss_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size,)`): A
+        mask which is 1 for examples for which their answer loss is larger than the answer_loss_cutoff.
     """
     # <float32>[batch_size]
     expected_result = _calculate_expected_result(

tests/test_modeling_tapas.py

@@ -14,7 +14,6 @@
 # limitations under the License.
 
 
-
 import unittest
 
 import numpy as np
@@ -26,17 +25,16 @@ from .test_configuration_common import ConfigTester
 from .test_modeling_common import ModelTesterMixin, floats_tensor, ids_tensor, random_attention_mask
 
 
-
 if is_torch_available():
     import torch
 
     from transformers import (
         TAPAS_PRETRAINED_MODEL_ARCHIVE_LIST,
         TapasConfig,
-        TapasModel,
         TapasForMaskedLM,
-        TapasForSequenceClassification,
         TapasForQuestionAnswering,
+        TapasForSequenceClassification,
+        TapasModel,
     )
 
 
@@ -144,9 +142,7 @@ class TapasModelTester:
 
         token_type_ids = []
         for type_vocab_size in self.type_vocab_sizes:
-            token_type_ids.append(
-                ids_tensor(shape=[self.batch_size, self.seq_length], vocab_size=type_vocab_size)
-            )
+            token_type_ids.append(ids_tensor(shape=[self.batch_size, self.seq_length], vocab_size=type_vocab_size))
         token_type_ids = torch.stack(token_type_ids, dim=2)
 
         sequence_labels = None
@@ -219,8 +215,18 @@ class TapasModelTester:
         )
 
     def create_and_check_model(
-        self, config, input_ids, input_mask, token_type_ids, sequence_labels, token_labels, label_ids, answer,
-        numeric_values, numeric_values_scale, aggregation_labels
+        self,
+        config,
+        input_ids,
+        input_mask,
+        token_type_ids,
+        sequence_labels,
+        token_labels,
+        label_ids,
+        answer,
+        numeric_values,
+        numeric_values_scale,
+        aggregation_labels,
     ):
         model = TapasModel(config=config)
         model.to(torch_device)
@@ -232,8 +238,18 @@ class TapasModelTester:
         self.parent.assertEqual(result.pooler_output.shape, (self.batch_size, self.hidden_size))
 
     def create_and_check_for_masked_lm(
-        self, config, input_ids, input_mask, token_type_ids, sequence_labels, token_labels, label_ids, answer,
-        numeric_values, numeric_values_scale, aggregation_labels
+        self,
+        config,
+        input_ids,
+        input_mask,
+        token_type_ids,
+        sequence_labels,
+        token_labels,
+        label_ids,
+        answer,
+        numeric_values,
+        numeric_values_scale,
+        aggregation_labels,
     ):
         model = TapasForMaskedLM(config=config)
         model.to(torch_device)
@@ -242,22 +258,48 @@ class TapasModelTester:
         self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length, self.vocab_size))
 
     def create_and_check_for_question_answering(
-        self, config, input_ids, input_mask, token_type_ids, sequence_labels, token_labels, label_ids, answer,
-        numeric_values, numeric_values_scale, aggregation_labels
+        self,
+        config,
+        input_ids,
+        input_mask,
+        token_type_ids,
+        sequence_labels,
+        token_labels,
+        label_ids,
+        answer,
+        numeric_values,
+        numeric_values_scale,
+        aggregation_labels,
     ):
         model = TapasForQuestionAnswering(config=config)
         model.to(torch_device)
         model.eval()
-        result = model(input_ids, attention_mask=input_mask, token_type_ids=token_type_ids,
-                       label_ids=label_ids, answer=answer, numeric_values=numeric_values,
-                       numeric_values_scale=numeric_values_scale, aggregation_labels=aggregation_labels
+        result = model(
+            input_ids,
+            attention_mask=input_mask,
+            token_type_ids=token_type_ids,
+            label_ids=label_ids,
+            answer=answer,
+            numeric_values=numeric_values,
+            numeric_values_scale=numeric_values_scale,
+            aggregation_labels=aggregation_labels,
         )
         self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length))
         self.parent.assertEqual(result.logits_aggregation.shape, (self.batch_size, self.num_aggregation_labels))
 
     def create_and_check_for_sequence_classification(
-        self, config, input_ids, input_mask, token_type_ids, sequence_labels, token_labels, label_ids, answer,
-        numeric_values, numeric_values_scale, aggregation_labels
+        self,
+        config,
+        input_ids,
+        input_mask,
+        token_type_ids,
+        sequence_labels,
+        token_labels,
+        label_ids,
+        answer,
+        numeric_values,
+        numeric_values_scale,
+        aggregation_labels,
     ):
         config.num_labels = self.num_labels
         model = TapasForSequenceClassification(config)
@@ -375,6 +417,7 @@ class TapasModelTest(ModelTesterMixin, unittest.TestCase):
     # def test_large_inputs_in_fp16_dont_cause_overflow(self):
     #     pass
 
+
 # Below: tests for Tapas utilities, based on segmented_tensor_test.py of the original implementation.
 # These test the operations on segmented tensors.
 class TapasUtilitiesTest(unittest.TestCase):

tests/test_tokenization_tapas.py

@@ -0,0 +1,573 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import functools
+import os
+import unittest
+from typing import List, Tuple
+
+import pandas as pd
+
+import regex as re
+from transformers import AddedToken, PreTrainedTokenizerBase
+from transformers.testing_utils import require_tokenizers, slow
+from transformers.tokenization_tapas import (
+    VOCAB_FILES_NAMES,
+    BasicTokenizer,
+    TapasTokenizer,
+    WordpieceTokenizer,
+    _is_control,
+    _is_punctuation,
+    _is_whitespace,
+)
+
+from .test_tokenization_common import TokenizerTesterMixin, filter_non_english
+
+
+@require_tokenizers
+class TapasTokenizationTest(TokenizerTesterMixin, unittest.TestCase):
+
+    tokenizer_class = TapasTokenizer
+    test_rust_tokenizer = False
+    space_between_special_tokens = True
+    from_pretrained_filter = filter_non_english
+
+    def get_clean_sequence(
+        self, tokenizer: TapasTokenizer, empty_table: bool = False, add_special_tokens: bool = True
+    ) -> Tuple[pd.DataFrame, str, List[int]]:
+        if empty_table:
+            table = pd.DataFrame.from_dict({})
+        else:
+            data = {
+                "Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
+                "Age": ["56", "45", "59"],
+                "Number of movies": ["87", "53", "69"],
+                "Date of birth": ["18 december 1963", "11 november 1974", "6 may 1961"],
+            }
+            table = pd.DataFrame.from_dict(data)
+        query = "Which actor appeared in the least number of movies?"
+
+        inputs = tokenizer.encode(table, query, add_special_tokens=add_special_tokens)
+
+        return table, query, inputs
+
+    # def get_clean_sequence(self, tokenizer, with_prefix_space=False, max_length=20, min_length=5) -> Tuple[str, list]:
+    #     data = {
+    #         'Actors': ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
+    #         'Age': ["56", "45", "59"],
+    #         'Number of movies': ["87", "53", "69"],
+    #         'Date of birth': ["18 december 1963", "11 november 1974", "6 may 1961"]
+    #     }
+    #     table = pd.DataFrame.from_dict(data)
+    #     output_ids = tokenizer.encode(table, add_special_tokens=False, max_length=max_length)
+    #     output_txt = tokenizer.decode(output_ids)
+    #
+    #     return output_txt, output_ids
+
+    def setUp(self):
+        super().setUp()
+
+        vocab_tokens = [
+            "[UNK]",
+            "[CLS]",
+            "[SEP]",
+            "[PAD]",
+            "[MASK]",
+            "want",
+            "##want",
+            "##ed",
+            "wa",
+            "un",
+            "runn",
+            "##ing",
+            ",",
+            "low",
+            "lowest",
+        ]
+        self.vocab_file = os.path.join(self.tmpdirname, VOCAB_FILES_NAMES["vocab_file"])
+        with open(self.vocab_file, "w", encoding="utf-8") as vocab_writer:
+            vocab_writer.write("".join([x + "\n" for x in vocab_tokens]))
+
+    def get_input_output_texts(self, tokenizer):
+        input_text = "UNwant\u00E9d,running"
+        output_text = "unwanted, running"
+        return input_text, output_text
+
+    def test_full_tokenizer(self):
+        tokenizer = self.tokenizer_class(self.vocab_file)
+
+        tokens = tokenizer.tokenize("UNwant\u00E9d,running")
+        self.assertListEqual(tokens, ["un", "##want", "##ed", ",", "runn", "##ing"])
+        self.assertListEqual(tokenizer.convert_tokens_to_ids(tokens), [9, 6, 7, 12, 10, 11])
+
+    def test_rust_and_python_full_tokenizers(self):
+        if not self.test_rust_tokenizer:
+            return
+
+        tokenizer = self.get_tokenizer()
+        rust_tokenizer = self.get_rust_tokenizer()
+
+        sequence = "UNwant\u00E9d,running"
+
+        tokens = tokenizer.tokenize(sequence)
+        rust_tokens = rust_tokenizer.tokenize(sequence)
+        self.assertListEqual(tokens, rust_tokens)
+
+        ids = tokenizer.encode(sequence, add_special_tokens=False)
+        rust_ids = rust_tokenizer.encode(sequence, add_special_tokens=False)
+        self.assertListEqual(ids, rust_ids)
+
+        rust_tokenizer = self.get_rust_tokenizer()
+        ids = tokenizer.encode(sequence)
+        rust_ids = rust_tokenizer.encode(sequence)
+        self.assertListEqual(ids, rust_ids)
+
+        # With lower casing
+        tokenizer = self.get_tokenizer(do_lower_case=True)
+        rust_tokenizer = self.get_rust_tokenizer(do_lower_case=True)
+
+        sequence = "UNwant\u00E9d,running"
+
+        tokens = tokenizer.tokenize(sequence)
+        rust_tokens = rust_tokenizer.tokenize(sequence)
+        self.assertListEqual(tokens, rust_tokens)
+
+        ids = tokenizer.encode(sequence, add_special_tokens=False)
+        rust_ids = rust_tokenizer.encode(sequence, add_special_tokens=False)
+        self.assertListEqual(ids, rust_ids)
+
+        rust_tokenizer = self.get_rust_tokenizer()
+        ids = tokenizer.encode(sequence)
+        rust_ids = rust_tokenizer.encode(sequence)
+        self.assertListEqual(ids, rust_ids)
+
+    def test_chinese(self):
+        tokenizer = BasicTokenizer()
+
+        self.assertListEqual(tokenizer.tokenize("ah\u535A\u63A8zz"), ["ah", "\u535A", "\u63A8", "zz"])
+
+    def test_basic_tokenizer_lower(self):
+        tokenizer = BasicTokenizer(do_lower_case=True)
+
+        self.assertListEqual(
+            tokenizer.tokenize(" \tHeLLo!how  \n Are yoU?  "), ["hello", "!", "how", "are", "you", "?"]
+        )
+        self.assertListEqual(tokenizer.tokenize("H\u00E9llo"), ["hello"])
+
+    def test_basic_tokenizer_lower_strip_accents_false(self):
+        tokenizer = BasicTokenizer(do_lower_case=True, strip_accents=False)
+
+        self.assertListEqual(
+            tokenizer.tokenize(" \tHäLLo!how  \n Are yoU?  "), ["hällo", "!", "how", "are", "you", "?"]
+        )
+        self.assertListEqual(tokenizer.tokenize("H\u00E9llo"), ["h\u00E9llo"])
+
+    def test_basic_tokenizer_lower_strip_accents_true(self):
+        tokenizer = BasicTokenizer(do_lower_case=True, strip_accents=True)
+
+        self.assertListEqual(
+            tokenizer.tokenize(" \tHäLLo!how  \n Are yoU?  "), ["hallo", "!", "how", "are", "you", "?"]
+        )
+        self.assertListEqual(tokenizer.tokenize("H\u00E9llo"), ["hello"])
+
+    def test_basic_tokenizer_lower_strip_accents_default(self):
+        tokenizer = BasicTokenizer(do_lower_case=True)
+
+        self.assertListEqual(
+            tokenizer.tokenize(" \tHäLLo!how  \n Are yoU?  "), ["hallo", "!", "how", "are", "you", "?"]
+        )
+        self.assertListEqual(tokenizer.tokenize("H\u00E9llo"), ["hello"])
+
+    def test_basic_tokenizer_no_lower(self):
+        tokenizer = BasicTokenizer(do_lower_case=False)
+
+        self.assertListEqual(
+            tokenizer.tokenize(" \tHeLLo!how  \n Are yoU?  "), ["HeLLo", "!", "how", "Are", "yoU", "?"]
+        )
+
+    def test_basic_tokenizer_no_lower_strip_accents_false(self):
+        tokenizer = BasicTokenizer(do_lower_case=False, strip_accents=False)
+
+        self.assertListEqual(
+            tokenizer.tokenize(" \tHäLLo!how  \n Are yoU?  "), ["HäLLo", "!", "how", "Are", "yoU", "?"]
+        )
+
+    def test_basic_tokenizer_no_lower_strip_accents_true(self):
+        tokenizer = BasicTokenizer(do_lower_case=False, strip_accents=True)
+
+        self.assertListEqual(
+            tokenizer.tokenize(" \tHäLLo!how  \n Are yoU?  "), ["HaLLo", "!", "how", "Are", "yoU", "?"]
+        )
+
+    def test_basic_tokenizer_respects_never_split_tokens(self):
+        tokenizer = BasicTokenizer(do_lower_case=False, never_split=["[UNK]"])
+
+        self.assertListEqual(
+            tokenizer.tokenize(" \tHeLLo!how  \n Are yoU? [UNK]"), ["HeLLo", "!", "how", "Are", "yoU", "?", "[UNK]"]
+        )
+
+    def test_wordpiece_tokenizer(self):
+        vocab_tokens = ["[UNK]", "[CLS]", "[SEP]", "want", "##want", "##ed", "wa", "un", "runn", "##ing"]
+
+        vocab = {}
+        for (i, token) in enumerate(vocab_tokens):
+            vocab[token] = i
+        tokenizer = WordpieceTokenizer(vocab=vocab, unk_token="[UNK]")
+
+        self.assertListEqual(tokenizer.tokenize(""), [])
+
+        self.assertListEqual(tokenizer.tokenize("unwanted running"), ["un", "##want", "##ed", "runn", "##ing"])
+
+        self.assertListEqual(tokenizer.tokenize("unwantedX running"), ["[UNK]", "runn", "##ing"])
+
+    def test_is_whitespace(self):
+        self.assertTrue(_is_whitespace(" "))
+        self.assertTrue(_is_whitespace("\t"))
+        self.assertTrue(_is_whitespace("\r"))
+        self.assertTrue(_is_whitespace("\n"))
+        self.assertTrue(_is_whitespace("\u00A0"))
+
+        self.assertFalse(_is_whitespace("A"))
+        self.assertFalse(_is_whitespace("-"))
+
+    def test_is_control(self):
+        self.assertTrue(_is_control("\u0005"))
+
+        self.assertFalse(_is_control("A"))
+        self.assertFalse(_is_control(" "))
+        self.assertFalse(_is_control("\t"))
+        self.assertFalse(_is_control("\r"))
+
+    def test_is_punctuation(self):
+        self.assertTrue(_is_punctuation("-"))
+        self.assertTrue(_is_punctuation("$"))
+        self.assertTrue(_is_punctuation("`"))
+        self.assertTrue(_is_punctuation("."))
+
+        self.assertFalse(_is_punctuation("A"))
+        self.assertFalse(_is_punctuation(" "))
+
+    def test_clean_text(self):
+        tokenizer = self.get_tokenizer()
+        # rust_tokenizer = self.get_rust_tokenizer()
+
+        # Example taken from the issue https://github.com/huggingface/tokenizers/issues/340
+        self.assertListEqual([tokenizer.tokenize(t) for t in ["Test", "\xad", "test"]], [["[UNK]"], [], ["[UNK]"]])
+
+        # self.assertListEqual(
+        #     [rust_tokenizer.tokenize(t) for t in ["Test", "\xad", "test"]], [["[UNK]"], [], ["[UNK]"]]
+        # )
+
+    @slow
+    def test_sequence_builders(self):
+        tokenizer = self.tokenizer_class.from_pretrained("tapas-base-uncased")
+
+        text = tokenizer.encode("sequence builders", add_special_tokens=False)
+        text_2 = tokenizer.encode("multi-sequence build", add_special_tokens=False)
+
+        encoded_sentence = tokenizer.build_inputs_with_special_tokens(text)
+        encoded_pair = tokenizer.build_inputs_with_special_tokens(text, text_2)
+
+        assert encoded_sentence == [101] + text + [102]
+        assert encoded_pair == [101] + text + [102] + text_2 + [102]
+
+    def test_offsets_with_special_characters(self):
+        for tokenizer, pretrained_name, kwargs in self.tokenizers_list:
+            with self.subTest("{} ({})".format(tokenizer.__class__.__name__, pretrained_name)):
+                tokenizer_r = self.rust_tokenizer_class.from_pretrained(pretrained_name, **kwargs)
+
+                sentence = f"A, naïve {tokenizer_r.mask_token} AllenNLP sentence."
+                tokens = tokenizer_r.encode_plus(
+                    sentence,
+                    return_attention_mask=False,
+                    return_token_type_ids=False,
+                    return_offsets_mapping=True,
+                    add_special_tokens=True,
+                )
+
+                do_lower_case = tokenizer_r.do_lower_case if hasattr(tokenizer_r, "do_lower_case") else False
+                expected_results = (
+                    [
+                        ((0, 0), tokenizer_r.cls_token),
+                        ((0, 1), "A"),
+                        ((1, 2), ","),
+                        ((3, 5), "na"),
+                        ((5, 6), "##ï"),
+                        ((6, 8), "##ve"),
+                        ((9, 15), tokenizer_r.mask_token),
+                        ((16, 21), "Allen"),
+                        ((21, 23), "##NL"),
+                        ((23, 24), "##P"),
+                        ((25, 33), "sentence"),
+                        ((33, 34), "."),
+                        ((0, 0), tokenizer_r.sep_token),
+                    ]
+                    if not do_lower_case
+                    else [
+                        ((0, 0), tokenizer_r.cls_token),
+                        ((0, 1), "a"),
+                        ((1, 2), ","),
+                        ((3, 8), "naive"),
+                        ((9, 15), tokenizer_r.mask_token),
+                        ((16, 21), "allen"),
+                        ((21, 23), "##nl"),
+                        ((23, 24), "##p"),
+                        ((25, 33), "sentence"),
+                        ((33, 34), "."),
+                        ((0, 0), tokenizer_r.sep_token),
+                    ]
+                )
+
+                self.assertEqual(
+                    [e[1] for e in expected_results], tokenizer_r.convert_ids_to_tokens(tokens["input_ids"])
+                )
+                self.assertEqual([e[0] for e in expected_results], tokens["offset_mapping"])
+
+    def test_add_special_tokens(self):
+        tokenizers: List[TapasTokenizer] = self.get_tokenizers(do_lower_case=False)
+        for tokenizer in tokenizers:
+            with self.subTest(f"{tokenizer.__class__.__name__}"):
+                input_table, input_query, ids = self.get_clean_sequence(
+                    tokenizer, empty_table=True, add_special_tokens=False
+                )
+
+                special_token = "[SPECIAL_TOKEN]"
+
+                tokenizer.add_special_tokens({"cls_token": special_token})
+                encoded_special_token = tokenizer.encode(input_table, special_token, add_special_tokens=False)
+                self.assertEqual(len(encoded_special_token), 1)
+
+                text = tokenizer.decode(ids + encoded_special_token, clean_up_tokenization_spaces=False)
+                encoded = tokenizer.encode(input_table, text, add_special_tokens=False)
+
+                input_encoded = tokenizer.encode(input_table, input_query, add_special_tokens=False)
+                special_token_id = tokenizer.encode(input_table, special_token, add_special_tokens=False)
+                self.assertEqual(encoded, input_encoded + special_token_id)
+
+                decoded = tokenizer.decode(encoded, skip_special_tokens=True)
+                self.assertTrue(special_token not in decoded)
+
+    def test_add_tokens_tokenizer(self):
+        tokenizers: List[TapasTokenizer] = self.get_tokenizers(do_lower_case=False)
+        for tokenizer in tokenizers:
+            with self.subTest(f"{tokenizer.__class__.__name__}"):
+                input_table, input_query, ids = self.get_clean_sequence(tokenizer, empty_table=True)
+                vocab_size = tokenizer.vocab_size
+                all_size = len(tokenizer)
+
+                self.assertNotEqual(vocab_size, 0)
+
+                # We usually have added tokens from the start in tests because our vocab fixtures are
+                # smaller than the original vocabs - let's not assert this
+                # self.assertEqual(vocab_size, all_size)
+
+                new_toks = ["aaaaa bbbbbb", "cccccccccdddddddd"]
+                added_toks = tokenizer.add_tokens(new_toks)
+                vocab_size_2 = tokenizer.vocab_size
+                all_size_2 = len(tokenizer)
+
+                self.assertNotEqual(vocab_size_2, 0)
+                self.assertEqual(vocab_size, vocab_size_2)
+                self.assertEqual(added_toks, len(new_toks))
+                self.assertEqual(all_size_2, all_size + len(new_toks))
+
+                tokens = tokenizer.encode(
+                    input_table, "aaaaa bbbbbb low cccccccccdddddddd l", add_special_tokens=False
+                )
+
+                self.assertGreaterEqual(len(tokens), 4)
+                self.assertGreater(tokens[0], tokenizer.vocab_size - 1)
+                self.assertGreater(tokens[-2], tokenizer.vocab_size - 1)
+
+                new_toks_2 = {"eos_token": ">>>>|||<||<<|<<", "pad_token": "<<<<<|||>|>>>>|>"}
+                added_toks_2 = tokenizer.add_special_tokens(new_toks_2)
+                vocab_size_3 = tokenizer.vocab_size
+                all_size_3 = len(tokenizer)
+
+                self.assertNotEqual(vocab_size_3, 0)
+                self.assertEqual(vocab_size, vocab_size_3)
+                self.assertEqual(added_toks_2, len(new_toks_2))
+                self.assertEqual(all_size_3, all_size_2 + len(new_toks_2))
+
+                tokens = tokenizer.encode(
+                    input_table,
+                    ">>>>|||<||<<|<< aaaaabbbbbb low cccccccccdddddddd <<<<<|||>|>>>>|> l",
+                    add_special_tokens=False,
+                )
+
+                self.assertGreaterEqual(len(tokens), 6)
+                self.assertGreater(tokens[0], tokenizer.vocab_size - 1)
+                self.assertGreater(tokens[0], tokens[1])
+                self.assertGreater(tokens[-2], tokenizer.vocab_size - 1)
+                self.assertGreater(tokens[-2], tokens[-3])
+                self.assertEqual(tokens[0], tokenizer.eos_token_id)
+                self.assertEqual(tokens[-2], tokenizer.pad_token_id)
+
+    @require_tokenizers
+    def test_encode_decode_with_spaces(self):
+        tokenizers = self.get_tokenizers(do_lower_case=False)
+        for tokenizer in tokenizers:
+            with self.subTest(f"{tokenizer.__class__.__name__}"):
+                input_table, input_query, ids = self.get_clean_sequence(tokenizer, empty_table=True)
+
+                # new_toks = ["[ABC]", "[DEF]"]  # TODO(thom) add this one back when Rust toks are ready: , "GHI IHG"]
+                new_toks = [AddedToken("[ABC]", normalized=False), AddedToken("[DEF]", normalized=False)]
+                tokenizer.add_tokens(new_toks)
+                input = "[ABC][DEF][ABC][DEF]"  # TODO(thom) add back cf above: "[ABC] [DEF] [ABC] GHI IHG [DEF]"
+                if self.space_between_special_tokens:
+                    output = "[ABC] [DEF] [ABC] [DEF]"
+                else:
+                    output = input
+                encoded = tokenizer.encode(input_table, input, add_special_tokens=False)
+                decoded = tokenizer.decode(encoded, spaces_between_special_tokens=self.space_between_special_tokens)
+                self.assertIn(decoded, [output, output.lower()])
+
+    def test_encode_plus_with_padding(self):
+        tokenizers = self.get_tokenizers(do_lower_case=False)
+        for tokenizer in tokenizers:
+            with self.subTest(f"{tokenizer.__class__.__name__}"):
+                input_table, input_query, ids = self.get_clean_sequence(tokenizer, empty_table=True)
+                sequence = "Sequence"
+
+                # check correct behaviour if no pad_token_id exists and add it eventually
+                self._check_no_pad_token_padding(tokenizer, sequence)
+
+                padding_size = 10
+                padding_idx = tokenizer.pad_token_id
+                token_type_padding_idx = tokenizer.pad_token_type_id
+
+                encoded_sequence = tokenizer.encode_plus(input_table, sequence, return_special_tokens_mask=True)
+                input_ids = encoded_sequence["input_ids"]
+                special_tokens_mask = encoded_sequence["special_tokens_mask"]
+                sequence_length = len(input_ids)
+
+                # Test 'longest' and 'no_padding' don't do anything
+                tokenizer.padding_side = "right"
+
+                not_padded_sequence = tokenizer.encode_plus(
+                    input_table,
+                    sequence,
+                    padding=True,
+                    return_special_tokens_mask=True,
+                )
+                not_padded_input_ids = not_padded_sequence["input_ids"]
+
+                not_padded_special_tokens_mask = not_padded_sequence["special_tokens_mask"]
+                not_padded_sequence_length = len(not_padded_input_ids)
+
+                assert sequence_length == not_padded_sequence_length
+                assert input_ids == not_padded_input_ids
+                assert special_tokens_mask == not_padded_special_tokens_mask
+
+                not_padded_sequence = tokenizer.encode_plus(
+                    input_table,
+                    sequence,
+                    padding=False,
+                    return_special_tokens_mask=True,
+                )
+                not_padded_input_ids = not_padded_sequence["input_ids"]
+
+                not_padded_special_tokens_mask = not_padded_sequence["special_tokens_mask"]
+                not_padded_sequence_length = len(not_padded_input_ids)
+
+                assert sequence_length == not_padded_sequence_length
+                assert input_ids == not_padded_input_ids
+                assert special_tokens_mask == not_padded_special_tokens_mask
+
+                # Test right padding
+                tokenizer.padding_side = "right"
+
+                right_padded_sequence = tokenizer.encode_plus(
+                    input_table,
+                    sequence,
+                    max_length=sequence_length + padding_size,
+                    padding="max_length",
+                    return_special_tokens_mask=True,
+                )
+                right_padded_input_ids = right_padded_sequence["input_ids"]
+
+                right_padded_special_tokens_mask = right_padded_sequence["special_tokens_mask"]
+                right_padded_sequence_length = len(right_padded_input_ids)
+
+                assert sequence_length + padding_size == right_padded_sequence_length
+                assert input_ids + [padding_idx] * padding_size == right_padded_input_ids
+                assert special_tokens_mask + [1] * padding_size == right_padded_special_tokens_mask
+
+                # Test left padding
+                tokenizer.padding_side = "left"
+                left_padded_sequence = tokenizer.encode_plus(
+                    input_table,
+                    sequence,
+                    max_length=sequence_length + padding_size,
+                    padding="max_length",
+                    return_special_tokens_mask=True,
+                )
+                left_padded_input_ids = left_padded_sequence["input_ids"]
+                left_padded_special_tokens_mask = left_padded_sequence["special_tokens_mask"]
+                left_padded_sequence_length = len(left_padded_input_ids)
+
+                assert sequence_length + padding_size == left_padded_sequence_length
+                assert [padding_idx] * padding_size + input_ids == left_padded_input_ids
+                assert [1] * padding_size + special_tokens_mask == left_padded_special_tokens_mask
+
+                if "token_type_ids" in tokenizer.model_input_names:
+                    token_type_ids = encoded_sequence["token_type_ids"]
+                    left_padded_token_type_ids = left_padded_sequence["token_type_ids"]
+                    right_padded_token_type_ids = right_padded_sequence["token_type_ids"]
+
+                    assert (
+                        token_type_ids + [[token_type_padding_idx] * 7] * padding_size == right_padded_token_type_ids
+                    )
+                    assert [[token_type_padding_idx] * 7] * padding_size + token_type_ids == left_padded_token_type_ids
+
+                if "attention_mask" in tokenizer.model_input_names:
+                    attention_mask = encoded_sequence["attention_mask"]
+                    right_padded_attention_mask = right_padded_sequence["attention_mask"]
+                    left_padded_attention_mask = left_padded_sequence["attention_mask"]
+
+                    assert attention_mask + [0] * padding_size == right_padded_attention_mask
+                    assert [0] * padding_size + attention_mask == left_padded_attention_mask
+
+    def test_internal_consistency(self):
+        tokenizers = self.get_tokenizers()
+        for tokenizer in tokenizers:
+            with self.subTest(f"{tokenizer.__class__.__name__}"):
+                input_table, input_query, ids = self.get_clean_sequence(tokenizer, empty_table=True)
+                input_text, output_text = self.get_input_output_texts(tokenizer)
+
+                tokens = tokenizer.tokenize(input_text)
+                ids = tokenizer.convert_tokens_to_ids(tokens)
+                ids_2 = tokenizer.encode(input_table, input_text, add_special_tokens=False)
+                self.assertListEqual(ids, ids_2)
+
+                tokens_2 = tokenizer.convert_ids_to_tokens(ids)
+                self.assertNotEqual(len(tokens_2), 0)
+                text_2 = tokenizer.decode(ids)
+                self.assertIsInstance(text_2, str)
+
+                self.assertEqual(text_2, output_text)
+
+    def test_mask_output(self):
+        tokenizers = self.get_tokenizers(fast=False, do_lower_case=False)
+        for tokenizer in tokenizers:
+            with self.subTest(f"{tokenizer.__class__.__name__}"):
+                input_table, input_query, ids = self.get_clean_sequence(tokenizer)
+
+                if (
+                    tokenizer.build_inputs_with_special_tokens.__qualname__.split(".")[0] != "PreTrainedTokenizer"
+                    and "token_type_ids" in tokenizer.model_input_names
+                ):
+                    information = tokenizer.encode_plus(input_table, input_query, add_special_tokens=True)
+                    sequences, mask = information["input_ids"], information["token_type_ids"]
+                    self.assertEqual(len(sequences), len(mask))