Subspace-PnP: A Geometric Constraint Loss for Mutual Assistance of Depth and Optical Flow Estimation.

Unsupervised optical flow and stereo depth estimation are two fundamental tasks in computer vision. Current studies (Tosi et al., in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4654–4665, 2020; Ranjan et al., in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12240–12249, 2019; Wang et al., in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8071–8081, 2019; Yin and Shi, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1983–1992, 2018) demonstrate that jointly learning networks for optical flow and stereo depth estimation via the geometric constraints can mutually benefit the two tasks and in turn yield large accuracy improvements. However, most of these methods generate geometric constraints based on estimated camera pose, which are not applicable to scenarios with moving objects that have different motions from the camera. In addition, errors of estimated camera pose would yield inaccurate constraints for the two tasks. In this paper, we propose a novel and universal geometric loss function, named Subspace-PnP, which is based on the Perspective-n-Points (PnP) and union-of-subspaces theory (Ji et al., in: IEEE Winter conference on applications of computer vision, pp 461–468, 2014) to jointly estimate the optical flow and stereo depth. The construction of Subspace-PnP dose not rely on the camera pose, but implicitly contains information of camera pose and motions of all moving objects. Our experiments show that the Subspace-PnP loss can mutually guide the estimation of optical flow and depth, enabling better robustness and greater accuracy even in dynamic scenes. In addition, we propose a motion-occlusion simulation method to handle occlusions caused by moving objects in optical flow estimation, which in turn can yield further performance improvement. Our method achieves the state-of-the-art performance for joint optical flow and stereo depth estimation on the KITTI 2012 and KITTI 2015 benchmarks. [ABSTRACT FROM AUTHOR]