Most existing structural health monitoring (SHM) techniques are based on measured acceleration data. Such practice, however, is highly expensive to operate, mainly due to cumbersome, time-consuming and expensive installation of sensors and their data acquisition systems. As an emerging noncontact method, the vision-based displacement sensor systems have attracted significant research interests and offered a promising alternative to the conventional sensors for SHM. However, most existing vision-based sensors require physical access to the structure to install a predesigned target panel, which has a higher contrast and thus is easier to track. Besides, most studies are carried out in controlled laboratory environments. The accuracy and robustness of vision sensors in the outdoor field conditions have not been fully investigated. It is also noted that current researches are mainly focusing on the measurement performance evaluation of vision sensors, without discussing the use of the measured displacement data for SHM. This dissertation develops a high-precision vision sensor system for remote and real-time measurement of multipoint structural displacements by tracking natural targets on structural surfaces. Two sets of software packages are developed respectively based on two advanced template matching algorithms (i.e., the upsampled cross correlation and the orientation code matching) incorporated with different subpixel techniques. Comprehensive experiments, including laboratory shaking table tests and field bridge tests, are carried out to evaluate its performance. Satisfactory agreements are observed between the displacements measured by the proposed vision sensor and those measured by high-performance reference displacement sensors. Moreover, this study examines the robustness of the vision sensor against ill environmental conditions such as dim light, background image disturbance and partial template occlusion. This dissertation further explores the potentials of the vision sensor for fast and inexpensive SHM applications, by demonstrating the usefulness of the displacement data for experimental modal analysis, finite element (FE) model updating, damage detection, etc. For a three-story frame structure, the modal analysis shows that the obtained natural frequencies and mode shapes from displacement measurements by using one camera match well with those by using four accelerometers. In fact, the vision sensor can achieve smoother mode shapes which would make damage localization more accurate, while the resolution of mode shapes from accelerometers is limited by the sensor number. This has been demonstrated from the damage detection result of beam structures based on the mode shape curvature (MSC) index. To address the needs for monitoring aging railway and highway bridges, coupled train-track-bridge and vehicle-bridge FE models are firstly developed to study the dynamic interactions between bridges and moving trains/vehicles. Subsequently, a time-domain model updating approach for railway bridges is proposed based on the in-situ measurement of the bridges’ dynamic displacement histories by the proposed vision sensor. This dissertation further proposes a bridge damage detection procedure that utilizes vehicle-induced displacement response and the MSC index without requiring prior knowledge about the traffic excitation.