Object must be selected manually and scale variation can't be handled with properly when using traditional algorithms. To overcome this problem, this paper proposed a new object tracking algorithm fused with optical flow detection and template matching. Firstly, it automatically detected and extracted object by combining the optical flow and the image segmentation results, accomplishing detection based tracking. It used the template matching technique for target locating when the result of the detection based tracking was unreliable, accomplishing matching based tracking. Finally, with the help of the template's auto update, adapted the target's changing in scale by the tracking window. The experimental results demonstrate that the proposed method can not only adapt object's scale variation, but acquires stable tracking results. Compared with three other algorithms, the proposed method has advantages in automatic detection and scale self-adaptation. [ABSTRACT FROM AUTHOR]