Summary

Deep learning on tabular data faces three challenges:

1. rotational variance -- the order of columns should not matter.
2. large data demand -- DNNs have a larger hypothesis space, and require more training data compared to shallow algorithms.
3. over-smooth solution -- DNNs tend to produce overly smooth solutions. i.e. when faced with irregular decision boundaries, the learning algorithms suffer (as pointed out by Grinsztajn et. al

Approach

Semi-Permiable Attention

The authors propose to add a mask to the attention score matrix such that the less important features do not influence more important features, but the more important features can influence the less important features.

where $\oplus$ denotes element-wise addition and $\oplus M$ is the proposed change to vanilla MHSA. $M \in \mathbb{R}^{f \times f}$ is a fixed mask, where

where $I(\bf{f}_i)$ is the importance of the $i$-th feature. In other words, this terms means that less informative features may use information from more informative features (case 0), but the opposite is blocked.

Interpolation-based data-augmentation

HID-mix

Given two samples $z_1^{(0)}, z_2^{(0)} \in \mathbb{R}^{f \times d}$ and their labels $y_1, y_2$, a new sample can be formed by mixing the embedding dimensions of $z_1^{(0)}$ and $z_2^{(0)}$:

where $S_H \in \{0,1\}^{f \times d}$ is a stack of binary masks $s_h:S_H = [s_h, s_h,..., s_h ]^T$, where $\sum s_h = \left\lfloor \lambda_H \cdot d \right\rfloor$ for each row vector $s_h$, and $\mathbb{1}$ is a $f \times d$ matrix of $1$s. In other words, $S_H$ masks out $\left\lfloor \lambda_H \cdot d \right\rfloor$ entries of each row.

FEAT-mix

Instead of mixing the embedding, FEAT-mix mixes the features given two samples $x_1, x_2 \in \mathbb{R}^{f}$ and their labels $y_1, y_2$, a new sample can be formed by mixing the features of $x_1$ and $x_2$:

where $s_F \in \{0,1\}^{f}$ is a binary mask vector where $\sum s_F = \left\lfloor \lambda_F \cdot f \right\rfloor$, $\mathbb{1}_F$ is a $f$ dimensional vector of $1$s, and $\Lambda$ is a scalar.

If we set $\Lambda = \lambda_F$, this equivalent to cutmix¹.

To differentiate, the authors introduce the usage of feature importance in the label weighting as follows:

where $s_F^{(i)}$ is the $i$-th element of $s_F$, and $I(\bf{f}_i)$ is the importance of the $i$-th feature. Similarly to the SPA module, the mutual information is what the authors appear to use.

Attentive FFNs

Finally, the authors propose to replace the 2-layer FFN module at the end of the transformer block with a 2-layer Gated Linear Unit (GLU) module instead, like the following:

where $\odot$ is element-wise multiplication and the first term acts as the gate.

In addition, the authors replace the linear embedding layer with similar GLU setup as well, which used to be $z_i = \bf{f}_i W_{i,1} + b_{i,1}$, into $z_i = \text{tanh}(\bf{f}_i W_{i,1} + b_{i,1}) \odot \bf{f}_i W_{i,2} + b_{i,2}$.

However, why they do this is not very clearly motivated.

Findings

• Excelformer works well on both small and large datasets!
  • While other models need HPO to be competitive, Excelformer is competitive without HPO.
• Both data augmentation methods are effective.

Resources

• Github