Improving the metric for evaluating cnns in SAR ATR applications by saliency maps

The CNNs manifest outstanding performance in SAR ATR applications. The most widely used means for evaluation is to test them on a separate testing set. However, as there exists a strong correlation between the working conditions of the training and the testing sets, the reliability of the method degrades. We revealed the problem by training and testing the models with pure clutters, and found that the model could still achieve a high accuracy. Furthermore, we noticed that some models showed approximately the same performance when the working condition was altered, while others suffered significant degradation. Analysis of the two types of models were conducted via saliency maps. Finally, a new inner feature ratio (IFR) metric for evaluation was proposed. Results demonstrated that the new metric could effectively reject those with false high accuracies, and help choose the robust ones.