Cross-model convolutional neural network for multiple modality data representation.

A novel data representation method of convolutional neural network (CNN) is proposed in this paper to represent data of different modalities. We learn a CNN model for the data of each modality to map the data of different modalities to a common space and regularize the new representations in the common space by a cross-model relevance matrix. We further impose that the class label of data points can also be predicted from the CNN representations in the common space. The learning problem is modeled as a minimization problem, which is solved by an augmented Lagrange method with updating rules of Alternating direction method of multipliers. The experiments over benchmark of sequence data of multiple modalities show its advantage. [ABSTRACT FROM AUTHOR]