AI可解释性 I | 对抗样本（Adversarial Sample）论文导读- Part I

• 单个高层神经元和多个高层神经元的线性组合之间并无差别，即表示语义信息的是高层神经元的空间而非某个具体的神经元
  

  there is no distinction between individual high level units and random linear combinations of high level units, ..., it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks.
  

  
  
• 神经网络的输入-输出之间的映射很大程度是不连续的，可以通过对样本施加难以觉察的噪声扰动（perturbation）最大化网络预测误差以使得网络错误分类，并且可以证明这种扰动并不是一种随机的学习走样（random artifact of learning），可以应用在不同数据集训练的不同结构的神经网络
  

  we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. Specifically, we find that we can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.
  

  
  

This suggest that the natural basis is not better than a random basis in for inspecting the properties of \(\phi(x)\). Moreover, it puts into question the notion that neural networks disentangle variation factors across coordinates.

In other words, it is possible for the output unit to assign non-significant (and, presumably, non-epsilon) probabilities to regions of the input space that contain no training examples in their vicinity.

• 对于文章研究的所有网络（包括MNIST、QuocNet、AlexNet），针对每个样本，始终能够生成与原始样本极其相似、视觉上无法区分的对抗样本，且这些样本均被原网络误分类。
  
• 跨模型的泛化性：当使用不同超参数（如层数、正则化项或初始权重）从头训练网络时，仍有相当比例的对抗样本会被误分类。
  
• 跨训练集的泛化性：在完全不同的训练集上从头训练的网络，同样会误分类相当数量的对抗样本。
  

• For all the networks we studied (MNIST, QuocNet [10], AlexNet [9]), for each sample, we always manage to generate very close, visually indistinguishable, adversarial examples that are misclassified by the original network (see figure 5 for examples).
  
• Cross model generalization: a relatively large fraction of examples will be misclassified by networks trained from scratch with different hyper-parameters (number of layers, regularization or initial weights).
  
• Cross training-set generalization a relatively large fraction of examples will be misclassified by networks trained from scratch trained on a disjoint training set.
  

A subtle, but essential detail is that adversarial examples are generated for each layer output and are used to train all the layers above. Adversarial examples for the higher layers seem to be more useful than those on the input or lower layers.

Still, this experiment leaves open the question of dependence over the training set. Does the hardness of the generated examples rely solely on the particular choice of our training set as a sample or does this effect generalize even to models trained on completely different training sets?

Uzuki评论：这个性质为对抗攻击的迁移性（transferability）提供了保证，那些无法得知内部结构黑盒网络，可以通过一个结构类似的白盒作为代理（surrogate model）生成对抗样本。

The intriguing conclusion is that the adversarial examples remain hard for models trained even on a disjoint training set, although their effectiveness decreases considerably.