Face age estimation has been widely used in video surveillance, human-computer interaction, market analysis, image processing analysis, and many fields. There are several problems that need to be solved in image-based face age estimation: (1) redundant information of age characteristics; (2) limitations of age estimation methods in solving age estimation problems; (3) the performance of age estimation models being also affected by gender factors. This paper proposes CA-XTree network. Firstly, features are extracted through the convolution layer and then combined with the local channel attention module to strengthen the ability of age feature information interaction between different channels. Secondly, extracted features are inputted into the recommendation score function to obtain the recommendation score, by combining the recommendation score with the gradient ascending regression tree. The lifting tree processed loss function is the mean square loss function, and the final age value is obtained by the leaf node. This paper improves state of the art for image classification on MORPH and CACD datasets. The advantage of our model is that it is easy to implement and has no excess memory overhead. In the age dataset CACD, the mean absolute error (MAE) has reached 4.535 and cumulative score (CS) has reached 63.53%, respectively. [ABSTRACT FROM AUTHOR]