Understanding adversarial robustness of deep neural networks via decision reliance

Adversarial robustness has become a major concern as machine learning models are increasingly deployed in high-risk and high-impact applications. Accordingly, various adversarial training strategies are proposed, making the model more robust under adversarial attack. However, similar to deep neural networks (DNNs) themselves, the mechanisms through which adversarial training strategies improve model robustness remain opaque. In this paper, we reveal how adversarial training alters the internal workings of deep neural networks by conducting neuron-wise decision reliance analysis. We find that adversarially vulnerable models predominantly rely on a small subset of predictive neurons while adversarially robust models tend to distribute their reliance across a broader range of neurons. We validate the relationship between decision reliance and adversarial robustness through comprehensive experiments across various models, training objectives, and attack scenarios. We observe that this relationship also holds for standard trained models, including those trained with Mixup or CutMix, which demonstrate improved performance against one-step adversarial attacks. Furthermore, we show that minimizing decision reliance leads to improved adversarial robustness. Our findings enrich the understanding of adversarially trained models and offer an interpretable and efficient approach to analyzing their internal mechanisms.