What Does CNN Shift Invariance Look Like?
A Visualization Study
| Jake Lee |
| Columbia University |
| Junfeng Yang |
| Columbia University |
| Zhangyang Wang |
| University of Texas at Austin |
Supplemental figures provided below
Abstract
Feature extraction with convolutional neural networks (CNNs) is a popular method to represent images for machine learning tasks. These representations seek to capture global image content, and ideally should be independent of geometric transformations. We focus on measuring and visualizing the shift invariance of extracted features from popular off-the-shelf CNN models. We present the results of three experiments comparing representations of millions of images with exhaustively shifted objects, examining both local invariance (within a few pixels) and global invariance (across the image frame). We conclude that features extracted from popular networks are not globally invariant, and that biases and artifacts exist within this variance. Additionally, we determine that anti-aliased models significantly improve local invariance, but do not impact global invariance. Finally, we provide a code repository for experiment reproduction, as well as a website to interact with our results.
Code & Data
- Experiment scripts
- [To Be Added]
- Interactive site source
- [GitHub Repo]
Paper
What Does CNN Shift Invariance Look Like? A Visualization Study
Accepted
[arXiv]
Figure 3 Additional Heatmaps: Large-Patch Dataset
Cosine similarity heatmaps of the large-patch dataset. Each pixel represents the average cosine similarity of different object patches at that shift location.
| Model | Top-Left | Top-Right | Center | Bottom-Left | Bottom-Right |
|---|---|---|---|---|---|
| AlexNet fc6 Off-the-Shelf |
 |
 |
 |
 |
 |
| AlexNet fc6 Anti-Aliased |
 |
 |
 |
 |
 |
| AlexNet fc7 Off-the-Shelf |
 |
 |
 |
 |
 |
| AlexNet fc7 Anti-Aliased |
 |
 |
 |
 |
 |
| AlexNet fc8 Off-the-Shelf |
 |
 |
 |
 |
 |
| AlexNet fc8 Anti-Aliased |
 |
 |
 |
 |
 |
| ResNet-50 Off-the-Shelf |
 |
 |
 |
 |
 |
| ResNet-50 Anti-Aliased |
 |
 |
 |
 |
 |
| MobileNetV2 Off-the-Shelf |
 |
 |
 |
 |
 |
| MobileNetV2 Anti-Aliased |
 |
 |
 |
 |
 |
Figure 4 Additional Heatmaps: Small-Patch Dataset
Cosine similarity heatmaps of the small-patch dataset. Each pixel represents the average cosine similarity of different object patches at that shift location.
| Model | Top-Left | Top-Right | Center | Bottom-Left | Bottom-Right |
|---|---|---|---|---|---|
| AlexNet fc6 Off-the-Shelf |
 |
 |
 |
 |
 |
| AlexNet fc6 Anti-Aliased |
 |
 |
 |
 |
 |
| AlexNet fc7 Off-the-Shelf |
 |
 |
 |
 |
 |
| AlexNet fc7 Anti-Aliased |
 |
 |
 |
 |
 |
| AlexNet fc8 Off-the-Shelf |
 |
 |
 |
 |
 |
| AlexNet fc8 Anti-Aliased |
 |
 |
 |
 |
 |
| ResNet-50 Off-the-Shelf |
 |
 |
 |
 |
 |
| ResNet-50 Anti-Aliased |
 |
 |
 |
 |
 |
| MobileNetV2 Off-the-Shelf |
 |
 |
 |
 |
 |
| MobileNetV2 Anti-Aliased |
 |
 |
 |
 |
 |