Exposition and Interpretation of the Topology of Neural Networks
Neural networks are powerful and widely used tools but their interpretability is often far from ideal. For this reason, it is interesting to develop methods to better understand their internal states. In this article we introduce the paper from Gabrielsson and Carlsson [1] in which they investigate how topological data analysis can be used to gain insight into the behavior and learning process of Convolutional Neural Networks.
Neural networks are powerful and widely used tools. They have already proven to be very efficient in the study of different types of data, whether it is images, texts, time series or many others. However, they are also highly complex and opaque tools whose interpretability is often far from ideal, which limits their usability.
One of their limitations is the difficulty of deducing their ability to generalize well to unseen data. Indeed, deep neural networks are prone to overfitting because they learn millions or even billions of parameters during training and models with so many parameters can easily overfit the training data since they have sufficient capacity to do so. This tendency to overfit can also make them vulnerable to the so-called “adversarial” behavior, where an imperceivable but carefully constructed nudge in the input can cause a CNN to fail to identify the content of an image. This phenomenon was discovered in 2014 by a group of researchers at Google and NYU. The corresponding inputs, called adversarial examples, are indistinguishable to the eye and are intentionally created to confuse the model into making an error. They are sort of optical illusions for machines. One of the most famous examples of an adversarial image was described in the paper of Goodfellow et al. in 2015 [2]. Starting with an image of a panda (left), the attacker adds small perturbations (middle) to this original image, and it results in the model recognizing the image as a gibbon with high confidence (right):
As we can see, we can clearly distinguish a panda in both images (left and right), in fact we are not even able to discern that some noise has been added to the original image! Yet, even if the disturbances are too subtle to be perceived by a human, they lead to performance degradation. This is particularly interesting to note that the original image is correctly classified as being a panda with about 60% confidence whereas the transformed image is incorrectly classified as a gibbon with almost 100% confidence!
For all these reasons, it is interesting to develop methods to gain some understanding of the internal states of neural networks. Given the very large number of neurons in a network, this poses a problem in data analysis and especially in unsupervised data analysis. It is therefore interesting to investigate how topological data analysis can be used to gain insight into the behavior and learning process of Convolutional Neural Networks.
- The first goal is to demonstrate that datasets derived from the weights of a CNN are organized into simple topological models.
- The second goal is to study the structures obtained in the first part and provide an interpretation of these topological models
Datasets
The subsequent figures and results are derived exclusively from convolutional neural networks trained on four different image datasets: MNIST, SVHN, CIFAR-10 and ImageNet. Although this work is applied to image datasets, the use of topological data analysis should also apply to explain the computations of many other types of neural networks.
Making Sense of a Convolutional Neural Network
In order to grasp the underlying behavior of a CNN, we need to understand its weight matrices. For this purpose, we consider a dataset X constructed out of the weights of the CNN where each data point is the weight matrix associated with a neuron in a fixed hidden layer. We collect data from all grids in this layer, and do this over many different trainings of the same network on the same training data. Once this dataset X is generated, we need to find a useful structure from a topological point of view. To do so, we only consider points of sufficiently high density. By performing a topological data analysis on this approximation of X, we can attempt to understand the behavior of the CNN.
■ Density Filtration
To determine the core subset X(k, p) of a set of points X, we perform a density filtration of these points based on a nearest neighbor estimate of the local density. It can be defined as follows:
dk(x) denotes the distance between x and its kᵗʰ nearest neighbor. This distance is inversely proportional to the density of points at x and so we define the density as the inverse of dk. The point cloud X is then approximated by the core subset X(k, p) where 0 < p ≤ 1, defined as the fraction p of points with the highest density. Thus, ρ(k, p, X) denotes a density filtration associated with parameters k and p applied to the point cloud X.
■ Mapper Method
We use a topological modeling method called Mapper [3]. It is based on the use of one, two or three real-valued functions on the data, called lenses, together with a metric on the dataset. Selecting overlapping coverings of the real line by intervals of the same length and overlap yields coverings of R, R², or R³ which allow us to group the data into bins, one for each set in the cover. For lenses we use the first and second principal components derived from PCA and then perform a clustering step using the Variance Normalized Euclidean metric which is a variant of the Euclidean distance that first normalizes each input data by dividing it by its variance. Since the intervals overlap, it is possible that the clusters attached to one bin overlap with the clusters attached to another bin. Finally, we establish a graph whose set of nodes is the collection of clusters we have thus defined and where an edge connects two clusters when they share at least one data point. We use the implementation of Mapper found in the Ayasdi software in which resolution specifies the number of bins and gain determines the overlap as follows: percent overlap = 1 − (1/gain). We specify Mapper by notation Mapper(resolution, gain).
Topological Analysis of Weight Spaces
To begin with, we train 100 CNNs of type M(64, 32, 64) on the MNIST dataset. The type we use here is defined in table I below (figure 4). While training our networks we use 40,000 mini-batch iterations each with a batch size of 128. The resulting test accuracy is almost 99.0%.
The 100 trained CNNs result in 64*100 = 6,400 spatial filters which are nine-dimensional points and which are mean-centered and normalized. We then use k-nearest-neighbor density filtration with k = 200 and p = 0.3 in order to derive a cloud of 1920 points to which we apply Mapper(resolution=30, gain=3) as described previously.
Studying the weights derived from the 1ˢᵗ convolutional layers of the 100 CNNs, we obtain the graph shown below, where the color of the nodes is determined by the number of points that the corresponding cluster contains, with red being the largest and blue the lowest:
Next to the graph, the explanation for the shape is also shown by labeling parts of the model with the mean of the corresponding set of weight matrices. From this graph we observe that:
a) The learned spatial filters are well approximated by the primary circle, the edges organize in a circle meaning that there are filters for each direction,
b) the circle is further supported by the persistent homology barcodes which show the presence of a persistent connected component and a persistent loop,
c) the model also over-represents the horizontal and vertical edges.
Moreover, these results were also robust to other convolutional neural network configurations including: M(64, 0, 64), M(64, 8, 512), M(64, 16, 512) and M(256, 32, 512).
In a second step, the study of the weights derived from the 1ˢᵗ convolutional layers but trained on CIFAR-10, gives the following topological model:
We note that the additional complexity of the CIFAR-10 dataset is shown in the horizontal and vertical lines and captures more detail than edges alone. It is consistent with the “three circle model” found in Gunnar Carlsson’s paper “On the Local Behavior of Spaces of Natural Images” [4]. It includes both the lines in the middle of the regions and the edges found.
As a first conclusion, we observe that the weight matrices derived from the training of a CNN, at least from the first convolutional layer, are organized into simple topological models that mimic the distribution of datasets present in natural images.
Understanding Weights Fluctuations
Now that we have established through topological data analysis that the weights of CNNs organize themselves into topological models that mimic the distribution of datasets occurring in natural images, we investigate further what happens during the process of neural network training.
The topological models obtained at different stages of CNN training on CIFAR-10 result in the following figure:
The figure above shows the topological models of the first and second layers (on the vertical axis) at different numbers of training iterations (on the horizontal axis). Note that for the 2nd layer spatial filters a density filtration was applied, while no density filtration was applied to the first layer.
We observe that the models obtained from the first convolutional layers seem to reproduce the circular pattern we discussed earlier around 400 iterations. After 500 iterations, the primary circle turns into a much more complex pattern that certainly includes the horizontal and vertical patches, but also a more complex pattern in the center of the model. This pattern is particularly visible at 1,000 iterations. On the other hand we also note that the four edges represented at 200 and 1,000 iterations seem to be relatively stable over many iterations.
The second row shows the models derived from the second convolutional layers. We observe that it is only after 2,000 iterations that the circular pattern starts to appear more clearly. Before that, we only observe a very weak circular pattern.
According to Gabrielsson and Carlsson, the second layer has taken over from the first layer, which has moved to capture more complex patches. This is a potential area of research for the future.
Topology of the Weights at Different Layers
So far, we have only described the results obtained from simple CNN architectures, however the method described in Gabrielsson and Carlsson’s paper also applies to deeper networks of different natures. For example, the study of the weight matrices obtained from convolutional layers of a pre-trained neural network on the ImageNet dataset gives the following figure:
We can see that the models coming from the second and third layers are very similar to the circular models mentioned before. The fourth layer also exhibits the same “primary circle” but includes lines in the middle of a background, these are the “secondary circles”. Beyond the fourth layer, more complex and interesting patterns are developed such as line crossings and “bull’s eyes”. Finally, the primary circle seems to remain the dominant structure in all layers, except in the last layer.
A Measure of Generality
As a reminder, one of the shortcomings of neural networks is the difficulty of deducing their ability to generalize well to unseen data. It is therefore interesting to study the topological structures of the learned weight matrices and try to understand if there exists a link between these structures and their ability to generalize to new data. Neural networks of type M(64, 32, 64) are then trained under three different circumstances on the MNIST dataset:
(i) Fixing the 1ˢᵗ conv layer to a perfect discretization of the primary circle,
(ii) Fixing the 1ˢᵗ conv layer to a random gaussian,
(iii) Without fixing anything as under regular circumstances.
Once trained and having achieved 99% accuracy on the MNIST test set, they are then evaluated on SVHN. The corresponding test accuracies are the following:
These results indicate that constraining an idealized version of the topological structure found in the data improves the ability of a CNN to generalize across different datasets.
Secondly, SVHN is a more diverse dataset (greater variety of fonts and styles) and this is why, while training a CNN on MNIST with a test accuracy of 99% yielded only 11% test accuracy on SVHN, training a CNN on SVHN to about 85% test accuracy yields 54% test accuracy on MNIST dataset:
Studying the graphs from these two experiments, we find that the weights of the first layer of the network learn a simpler and stronger primary circle when the network is trained on SVHN than on MNIST. Thus, these observations demonstrate that topological structure is indicative of a network’s ability to generalize between different datasets. Moreover, they are consistent with the hypothesis that a simpler topology implies better generalization abilities.
Conclusion
As a conclusion, we saw that datasets derived from the weights of a CNN are organized into simple topological models. We found that in some cases these models are strongly similar to the results obtained in the topological analysis of local patch datasets in natural images, namely primary and secondary circles, but that in other situations different phenomena occur.
Once we demonstrated that the information encoded in the weights of a CNN can be organized in terms of a topological data model, we studied these structures and provided an interpretation. We found that convolutional layer weights at depths from 1 to 13 learn simple global structures, and we also noted the change in simple structures during learning iterations. Analyzing the spatial filters of the convolutional layers we found a recurrence among all networks, all depths, and during learning, of a simple circle consisting of rotating edges, as well as a less recurrent unanticipated complex circle that combines lines, edges, and nonlinear patterns.
Finally we found that the topological structure is indicative of the ability of a neural network to generalize to unseen data: simpler topology implies better generalization abilities.
I hope you enjoyed this article! I strongly encourage you to read the paper by Gabrielsson and Carlsson which I found particularly interesting. Also, feel free to contact me if you have any questions or if you feel that additional explanations should be added to my article.
Thanks for reading! :)
References
[1] Exposition and Interpretation of the Topology of Neural Networks, Rickard Brüel Gabrielsson, Gunnar Carlsson
[2] Explaining and Harnessing Adversarial Examples, Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy
[3] Topological methods for the analysis of high dimensional data sets and 3D object recognition, Gurjeet Singh, Facundo Mémoli, Gunnar Carlsson
[4] On the Local Behavior of Spaces of Natural Images, Gunnar Carlsson Tigran Ishkhanov, Vin de Silva, Afra Zomorodian
