Simulations to benchmark time-varying connectivity methods for fMRI

There is a current interest in quantifying time-varying connectivity (TVC) based on neuroimaging data such as fMRI. Many methods have been proposed, and are being applied, revealing new insight into the brain’s dynamics. However, given that the ground truth for TVC in the brain is unknown, many concerns remain regarding the accuracy of proposed estimates. Since there exist many TVC methods it is difficult to assess differences in time-varying connectivity between studies. In this paper, we present tvc_benchmarker, which is a Python package containing four simulations to test TVC methods. Here, we evaluate five different methods that together represent a wide spectrum of current approaches to estimating TVC (sliding window, tapered sliding window, multiplication of temporal derivatives, spatial distance and jackknife correlation). These simulations were designed to test each method’s ability to track changes in covariance over time, which is a key property in TVC analysis. We found that all tested methods correlated positively with each other, but there were large differences in the strength of the correlations between methods. To facilitate comparisons with future TVC methods, we propose that the described simulations can act as benchmark tests for evaluation of methods. Using tvc_benchmarker researchers can easily add, compare and submit their own TVC methods to evaluate its performance.
Author summary Time-varying connectivity attempts to quantify the fluctuating covariance relationship between two or more regions through time. In recent years, it has become popular to do this with fMRI neuroimaging data. There have been many methods proposed to quantify time-varying connectivity, but very few attempts to systematically compare them. In this paper, we present tvc_benchmarker, which is a python package that consists of four simulations. The parameters of the data are justified on fMRI signal properties. Five different methods are evaluated in this paper, but other researchers can use tvc_benchmarker to evaluate their methodologies and their results can be submitted to be included in future reports. Methods are evaluated on their ability to track a fluctuating covariance parameter between time series. Of the evaluated methods, the jackknife correlation method performed the best at tracking a fluctuating covariance parameter in these four simulations.