Filtering Limited Automatic Vehicle Identification Data for Real-Time Path Travel Time Estimation Without Ground Truth

Automatic Vehicle Identification (AVI) technology has been widely used for real-time path travel time estimation. For a study path equipped with AVI sensors at both ends, the difference between the timestamps of vehicles entering and leaving the path is AVI data. In urban areas, there can be several alternative routes and vehicle entry/exit points for the study path. Consequently, invalid AVI data occur that fall outside the scope of the travel time of the study path. Some AVI technologies based on identification information of vehicles can match vehicles precisely. However, for cities like Hong Kong with concerns of privacy issues, only commercial vehicle data can be collected. Under this scenario, the resultant AVI data are accurate but with few valid samples in a relatively short time interval due to the unavailability of private car data. The estimation accuracy of path travel times on a real-time basis will then be affected significantly by the existence of invalid AVI data. In this paper, a novel unsupervised algorithm is proposed to filter out real-time invalid AVI data efficiently although there is no ground truth available for training purposes. It is tested and compared with other benchmark algorithms on two selected paths in the Hong Kong urban road network. It is found that the proposed unsupervised algorithm can still filter limited but accurate AVI data with satisfactory performance. Sensitivity tests with ground truth are also conducted with different sampling rates. Some insightful findings are given for filtering AVI data under various scenarios.