Mixed Dish Recognition With Contextual Relation and Domain Alignment

Mixed dish is a food category that contains differentdishes mixed in one plate, and is popular in Eastern andSoutheast Asia. Recognizing the individual dishes in a mixeddish image is important for health related applications, e.g. tocalculate the nutrition values of the dish. However, most existingmethods that focus on single dish classification are not applicableto the recognition of mixed dish images. The main challenge ofmixed dish recognition comes from three aspects: a wide rangeof dish types, the complex dish combination with severe overlapbetween different dishes and the large visual variances of samedish type caused by different cooking/cutting methods applied indifferent canteens. In order to tackle these problems, we proposethe contextual relation network that encodes the implicit andexplicit contextual relations among multiple dishes from region-level features and label-level co-occurrence respectively. Besides,to address the visual variances of dish instances from differentcanteens, we introduce the domain adaption networks to alignboth local and global features, and eliminating domain gaps ofdish features across different canteens. In addition, we collecta mixed dish image dataset containing 9,254 mixed dish imagesfrom 6 canteens in Singapore. Extensive experiments on both ourdataset and public one validate that our methods can achieve topperformance for localizing and recognizing multiple dishes andsolve the domain shift problem to a certain extent in mixed dishimages