TabTransformers
TabTransformer is a novel deep data modeling based architecture, for supervised and semi-supervised learning.The model converts embeddings of categorial features into robust contextual embeddings**. These contextual embeddings are also resilient to missing and noisy data.
TabTransformer: A Novel Deep Learning Architecture for Tabular Data
Abstract
TabTransformer is a novel deep data modeling based architecture, for supervised and semi-supervised learning.The model converts embeddings of categorial features into robust contextual embeddings. These contextual embeddings are also resilient to missing and noisy data.
The key takeaways from this paper are:
- The architecture is built upon self-attention based transformers
- On supervised datasets it achieves a 1. 0% mean AUC improvement and is on par with other true based ensemble models.
- On the semi-supervised setting it has around a 20% gain on the mean AUC
Introduction
Tabular Data is the most common type of data that humans come across on real world problems. Examples where we come across tabular data are recommender systems, online advertising and so on. Platforms like Kaggle and KDD are specifically designed to tackle the tabular data domain.
Currently, Gradient Boosted Decision Trees (GBDTs) are considered state-of-the-art models for tabular data. While deep learning architectures have revolutionized image and text domains, GBDTs still maintain their strong performance in tabular data. They offer competitive performance quickly and are easily interpretable, making them popular among ML practitioners.
GBDT however do have certain drawbacks:
- They are not suitable for continual training on streaming data and do not allow efficient end-to-end pipelines for image/text encoders that are needed for multimodality.
- On its basic form, it's not suitable for semi-supervised learning and this is due to the fact that the decision learners do not produce an efficient probability distribution
- State-of-the-art deep learning methods for handling noisy and missing data don't apply to them.
- The robustness of GBDT has also not been studied properly.
A classical model trained using gradient descent which is capable of learning end-to-end image/text encoders are MLPs. This is possible because the categorial data features are represented as learned parametric embeddings.
MLP's due to their shallow architecture and non-contextual embeddings have some limitations such as:
1. Neither the model nor the embeddings are interpretable 2. They are not robust to noisy or missing data 3. Their performance on semi-supervised learning is bad
Researchers have tried to bridge the gap between MLP's and tree based methods but they do not attend to the limitations caused by GBDT's and MLP's
It is also noted that on large scale real word type data, the tree based models outperform MLPs. This paper introduces Tab Transformers to address the limitations of MLP based architectures and bridge the gap between MLP's and tree based methods
The Tab Transformer model is evaluated on 15 publicly available datasets. It is based on the transformer architecture, converting categorical data features into contextual embeddings. This differs from NLP where words are encoded into a dense, lower-dimension space. While word2vec and context-free BERT provide word-level context , contextual embedding methods, primarily based on self-attention, have led to many state-of-the-art (SOTA) models in this domain.
Motivated by the success of transformers in NLP, the authors try to incorporate this in the tabular domain. TabTransformers basically uses self-attention based transformer layers to transform context free parametric embeddings to contextual embeddings. These embeddings and highly interpretable and robust to noisy and missing data
Highly correlated features, including feature pairs in the same column tend to have a smaller Euclidean distance. There is no such pattern observed in the context free embeddings
Existing deep learning architectures for tabular data primarily focus on supervised learning, with very few addressing unsupervised learning. State-of-the-art semi-supervised learning approaches from NLP and computer vision are challenging to adapt to the tabular domain. To address this, the authors leverage pre-training methodologies from language models to pre-train a semi-supervised transformer for Tab Transformer. The advantage of this is the fact that there are training phases: 1. A costly pre-training phase on unlabeled data 2. A lightweight fine-tuning phase on labeled data
This is different from the state of the art that needs one training phase with both labeled and unlabeled data. Breaking the training phase into 2 helps business that have to use the same dataset for multiple analysis.
The contributions of this paper are:
1. Introduce TabTransformer which can convert categorial tabular data to contextual embeddings. This in turn gives you performance better than MLPs and current SOTA deep learning models while being on par with GBDT methods
2. Investigate the contextual embeddings and compare them to parametric context free embeddings
3. Prove the robustness of the contextual embedding with respect to noisy and missing data.
4. Introduce a 2 phase training approach that beats state of the art methods in semi-supervised learning.
Tab Transformer
The TabTransfomer consists of a column embedding layer, a stack of N x transformer layers and a multi layer perception. Each transformer layer is made of a multi-head attention layer followed by a position wise feed forward neural network
The working of TabTransfomer:
Let (x,y)denote a feature target pair and x ≌ (X_cat,X_count).Here X_cat denotes the categorical features and X_count devotes the continuous features
- X_cat = {x1,x2, .... Xm} and these are then inputted into the column embedding layer to get parametric embedding of dimension ’d’
- Let e(Xi_)∈R for i ∈{1,…,m} denote the parametric embedding for the X_i categorical features
- By This we can denote E(X_cat)={e(X_1),e(X_2),…,e(X_m)} as the parametric embeddings outputted by the column embedding layer.
- These parametric embeddings e(X_cat)are the inputs to the first transformer layer. The output of the first transformer layer is the input to the second transformer layer and so on.
Using this approach, the parametric embeddings are converted into contextual embeddings, by the top transformer layer, by aggregating context from other layer embeddings.The sequence of transformer layers are denoted as a function f_o.
The function f_o takes e(X_cat) as input to produce {h1, h2... hm} as the outputs. The contextual embeddings are then concatenated with the the continuous features (X_cont)to form a rector of dimension (d x m + c) where 'c’ is the number of continuous features and m is the count of categorical features
This vector is then inputted into an MLP with an function denoted by ‘g'.
Let H be the cross entropy for classification tasks and MSE for regression tasks. The goal is to minimize L(x, y) to learn all the TabTransformer parameters Ф for the column embedding, ø for the transformer layers, Ψ for the MLP layers using all first order gradient methods for learning.
, R^(m x k), R^(m x v) where m is the number of embeddings inputted into the transformer and v in the dimensions of the key/ value matrices.
Every input embedding attends to all other embedding through a attention head which is computed as:
For each of the input embedding, the attention matrix A∈R^(m x n) calculates how much it attends to other embeddings and transforms the embeddings into contextual ones. The output of the attention head (contextual embeddings of dimension 'v') is projected into dimension 'd' using a fully connected layer. This 'd'-dimension embedding then passes through two layers of position-wise feed-forward layers: one doubles the dimension, and the other returns it to the original dimension.
Column Embedding Layer
For each categorical feature, which is basically a column, there exists an lookup table eϕ for each categorical feature column present. For an categorical feature with ‘p’ classes the lookup table or embedding table (p+1) vector embedding, where the extra rector is used to denote the missing class. The output of the lookup or embedding table in denoted as
C_ϕ=[C_ϕl,ω_ϕd−l]
Here C_ϕ in a unique identifier which is same for all the classes in the categorical feature column. The dimension of this unique identifier is a hyperparameter. While the unique identifier is the same, the actual class values are encoded in the W_a matrix.
The use of this "unique identifier" is specific to tabular data. In language models, an element-wise addition is performed between positional word embeddings and vector embeddings. However, tabular data lacks positional order, making this approach unsuitable. Other strategies explored for the column embedding layers involved fixing 'd' values and using element-wise addition between the unique identifier and feature-value specific embeddings, rather than concatenating them.
Pre-training the Embeddings
The contextual embeddings are trained end to end using labelled data. In cases where there are loads of unlabelled data and some labelled data, pre-training is introduced. During pre-training, the transformer layers are trained on unlabeled data, and then both the transformer and MLP layers are fine-tuned on labeled data. The loss function used during pre-training is the same as the final task.
There are 2 pretraining methods that the model is trained on:
-
Masked Language Modelling (MLM): MLM basically takes K% of the features from the or cat input features and masks them as missing. The transformer layers along with the column embedding layer is trained using a multi-class classifier based loss function to predict the original features from the masked features, using the contextual embeddings outputted by the last transformer layer.
-
Replaced Token Detection (RTD): Here instead of masking the features, the features are replaced with random values. The transformer along with the column embedding layers learns through a binary classifier loss function to predict whether the feature is replaced or not. The initial RTD approach uses an auxiliary generator to sample a subset of features from which the original feature would be replaced with. In language modelling situations, the number of tokens in the vocabulary are immense and hence detecting a uniform random token is much easier. In contrast to this:
-
Tabular data has much less number of classes within a categorial feature
-
Since each feature column has its own embedding table, the binary classifier is unique of each column rather than having a single shared one
-
Experiments
The Tab Transformer architecture was evaluated on 15 datasets from various repositories, including Kaggle and the AutoML challenge. The data split used was 65/15/20 for training, testing, and validation sets respectively. The number of classes across datasets ranged from 2 to 136. For semi-supervised learning, a proportion 'p' of training samples (50, 200, or 500) were marked as labeled, with the rest as unlabeled.
The Tab Transformer utilized a hidden embedding dimension of 32, 6 layers, and 8 attention heads. MLP layer sizes were set to {4×I,2×I}, where I is the input dimension.
Hyperparameter optimization (HPO) involved 20 HPO rounds for each validation set per model. Pre-training was exclusively applied in the semi-supervised setting, where significant benefits were observed; in supervised settings, the benefits were not significant.
Effectiveness of the Transformer Layers
The transformer layers are first compared with the baseline MLP. For this the transformer layers are removed and the output of this is compared to the original TabTransformer model. The dimensions of both the models for their categorial embedding is both set to 32. In the comparison over 15 datasets, the Tabiransformer architecture has an overall average gain of 1% AUC
The authors then take the contextual embeddings and plot their corresponding t-SNE graphs to visualize their similarity. In more detail, the categorial columns of the test data are made to pass through a fully trained TabTransformer model, from which the contextual embeddings are extracted from a certain layer of the transformer. The t-SNE algorithm is then used to plot the embedding as a 2D visualization.Through this we can see that semantically similar classes are closer to each other than classes that are not semantically similar.
To furthermore test the effectiveness of the transformer layers, the authors take the contextual embeddings of each transformer layer in the trained TabTransformer, which is formed by passing the column features of the test data. This is then concatenated with the continuous features of the dataset and then a logistic regression based MLP is made to predict the target variables of the test data using these as input.
For each dataset and each layer the average CV-Score is calculated and then plotted. As the graph shows an upward trend it proves that the contextual embeddings caused by the upper layer has better meaning. Furthermore the last value of each of these lines is close to 1 showcasing that the contextual embeddings are reliable
Here's the graph (Figure 3 from the paper) illustrating this point:
Robustness of TabTransformer
Noisy data:
To test the robustness of the TabTransformer architecture on noisy data. Randomly generated data was used to replace ‘p’ set of values from their irrespective column features. A fully trained TabTransformer is then used to make predictions of this data. As the noisy rate increases the TabTransformer performance also increases, proving it to be better than the standard MLP. In fact in one dataset the performance did not reduce at all which was attributed to the robustness of the contextual Embeddings.
Data with Missing values:
Similar to the noisy data experiment, 'y' samples of values were randomly selected and marked as missing, then fed into the Tab Transformer. The model can handle missing values in two ways:
- Using the average learned embeddings of all classes in the corresponding column.
- Using an additional embedding for each class of the missing value.
Since there weren't enough missing values in the training data, the first method (using average learned embeddings) was employed. The results were very similar to those observed with noisy data, further showcasing the Tab Transformer's robustness
Link to the github repo for the implementation : Paper Implementation
Credit
This implementation is inspired by and based on the official TabTransformer paper.All credit for the model architecture and its innovative concepts goes to the original authors: Xin Huang, Ashish Kunnath, and Ron Bodkin.