Recently, the number of surveillance video data has grown enormously since the popularity of the smart cities, which brings the difficulties to users for searching and analyzing the content of the videos. These video data are not limited to data anymore, which provide the effective information for criminal investigation systems, intrusion detection system, and many others. However, as the number of available cloud services increases, the problem of data discovery and selection arises. The semantic technology is an effective choice for enhancing the accurate of the data searching and analyzing process. In this paper, a semantic based cloud environment is proposed to facilitate the analyzing and searching process of surveillance video data. An architecture integrating ontology building, semantic annotation, and semantic search is proposed to leverage the semantic description of the video data to find them from concept-based level. A semantic intermediate layer which organizes the video data based on their semantic relations is given. Moreover, the proposed method is used in the intelligent transportation field, which shows the bright prospect of the proposed method in real applications. [ABSTRACT FROM AUTHOR]