Search

We develop E-variables for testing whether two or more data streams come from the same source or not, and more generally, whether the difference between the sources is larger than some minimal effect size. These E-variables lead to exact, nonasymptotic tests that remain safe, i.e., keep their type-I error guarantees, under flexible sampling scenarios such as optional stopping and continuation. In special cases our E-variables also have an optimal ‘growth’ property under the alternative. While the construction is generic, we illustrate it through the special case of k×2 contingency tables, i.e. k Bernoulli streams, allowing for the incorporation of different restrictions on the composite alternative. Comparison to p-value analysis in simulations and a real-world 2 × 2 contingency table example show that E-variables, through their flexibility, often allow for early stopping of data collection — thereby retaining similar power as classical methods — while also retaining the option of extending or combining data afterwards.

It is currently difficult to successfully choose the correct type of antidepressant for individual patients. To discover patterns in patient characteristics, treatment choices and outcomes, we performed retrospective Bayesian network analysis combined with natural language processing (NLP). This study was conducted at two mental healthcare facilities in the Netherlands. Adult patients admitted and treated with antidepressants between 2014 and 2020 were included. Outcome measures were antidepressant continuation, prescription duration and four treatment outcome topics: core complaints, social functioning, general well-being and patient experience, extracted through NLP of clinical notes. Combined with patient and treatment characteristics, Bayesian networks were constructed at both facilities and compared. Antidepressant choices were continued in 66% and 89% of antidepressant trajectories. Score-based network analysis revealed 28 dependencies between treatment choices, patient characteristics and outcomes. Treatment outcomes and prescription duration were tightly intertwined and interacted with antipsychotics and benzodiazepine co-medication. Tricyclic antidepressant prescription and depressive disorder were important predictors for antidepressant continuation. We show a feasible way of pattern discovery in psychiatry data, through combining network analysis with NLP. Further research should explore the found patterns in patient characteristics, treatment choices and outcomes prospectively, and the possibility of translating these into a tool for clinical decision support.

Introduction: We developed and tested a Bayesian network(BN) model to predict ECT remission for depression, with non-response as a secondary outcome. Methods: We performed a systematic literature search on clinically available predictors. We combined these predictors with variables from a dataset of clinical ECT trajectories (performed in the University Medical Center Utrecht) to create priors and train the BN. Temporal validation was performed in an independent sample. Results: The systematic literature search yielded three meta-analyses, which provided prior knowledge on outcome predictors. The clinical dataset consisted of 248 treatment trajectories in the training set and 44 trajectories in the test set at the same medical center. The AUC for the primary outcome remission estimated on an independent validation set was 0.686 (95%CI 0.513–0.859) (AUC values of 0.505 – 0.763 observed in 5-fold cross validation of the model within the train set). Accuracy 0.73 (balanced accuracy 0.67), sensitivity 0.55, specificity 0.79, after temporal validation in the independent sample. Prior literature information marginally reduced CI width. Discussion: A BN model comprised of prior knowledge and clinical data can predict remission of depression after ECT with reasonable performance. This approach can be used to make outcome predictions in psychiatry, and offers a methodological framework to weigh additional information, such as patient characteristics, symptoms and biomarkers. In time, it may be used to improve shared decision-making in clinical practice.

Sequential decision making significantly speeds up research and is more cost-effective compared to fixed-n methods. We present a method for sequential decision making for stratified count data that retains Type-I error guarantee or false discovery rate under optional stopping, using e-variables. We invert the method to construct stratified anytime-valid confidence sequences, where cross-talk between subpopulations in the data can be allowed during data collection to improve power. Finally, we combine information collected in separate subpopulations through pseudo-Bayesian averaging and switching to create effective estimates for the minimal, mean and maximal treatment effects in the subpopulations.

E-variables are tools for retaining type-I error guarantee with optional stopping. We extend E-variables for sequential two-sample tests to general null hypotheses and anytime-valid confidence sequences. We provide implementations for estimating risk difference, relative risk and odds-ratios in contingency tables.

We present tests for conditional independence of discrete variables that can be applied in a sequential setting with Type-I error probability guarantee. Power of these tests can be improved by incorporating hypothesized effect size or by sharing information between strata. Both scenarios are illustrated through simulations.

Background: Developing predictive models for precision psychiatry is challenging because of unavailability of the necessary data: extracting useful information from existing electronic health record (EHR) data is not straightforward, and available clinical trial datasets are often not representative for heterogeneous patient groups. The aim of this study was constructing a natural language processing (NLP) pipeline that extracts variables for building predictive models from EHRs. We specifically tailor the pipeline for extracting information on outcomes of psychiatry treatment trajectories, applicable throughout the entire spectrum of mental health disorders (“transdiagnostic”). Methods: A qualitative study into beliefs of clinical staff on measuring treatment outcomes was conducted to construct a candidate list of variables to extract from the EHR. To investigate if the proposed variables are suitable for measuring treatment effects, resulting themes were compared to transdiagnostic outcome measures currently used in psychiatry research and compared to the HDRS (as a gold standard) through systematic review, resulting in an ideal set of variables. To extract these from EHR data, a semi-rule based NLP pipeline was constructed and tailored to the candidate variables using Prodigy. Classification accuracy and F1-scores were calculated and pipeline output was compared to HDRS scores using clinical notes from patients admitted in 2019 and 2020. Results: Analysis of 34 questionnaires answered by clinical staff resulted in four themes defining treatment outcomes: symptom reduction, general well-being, social functioning and personalization. Systematic review revealed 242 different transdiagnostic outcome measures, with the 36-item Short-Form Survey for quality of life (SF36) being used most consistently, showing substantial overlap with the themes from the qualitative study. Comparing SF36 to HDRS scores in 26 studies revealed moderate to good correlations (0.62—0.79) and

We develop sequential A/B tests with strict Type-I error control under optional stopping. Our tests are based on E-variables, which recently have turned out be successful tools for anytime-valid inference. We introduce a general method for constructing E-variables that can be used for A/B testing in 2-sample streams. In contrast to earlier methods developed in the sequential testing literature, our approach is valid for both balanced and unbalanced experiments and allows for arbitrary, user-specified notions of effect size. The same method can be used to design anytime-valid confidence sequences to estimate effect sizes in data streams. With two Bernoulli streams as a running example, we illustrate the power of our A/B test and show that decisions can often be made earlier compared to classical methods, such as Fisher's exact test. We also illustrate the confidence sequences with two different notions of effect size: log odds ratio and difference in mean.

E variables are tools for designing tests that keep their type-I error guarantees under flexible sampling scenarios such as optional stopping and continuation. We extend the recently developed E variables for two-sample tests to general null hypotheses and the corresponding anytime-valid confidence sequences. Using the 2x2 contingency table (Bernoulli) setting as a running example, we provide simple implementations of these confidence sequences for linear and odds-ratio based effect size.

Functions to design and apply tests that are anytime valid. The functions can be used to design hypothesis tests in the prospective/randomised control trial setting or in the observational/retrospective setting. The resulting tests remain valid under both optional stopping and optional continuation. The current version includes safe t-tests and safe tests of two proportions. For details on the theory of safe tests, see Grunwald, de Heide and Koolen (2019) "Safe Testing" , for details on safe logrank tests see ter Schure, Perez-Ortiz, Ly and Grunwald (2020) "The Safe Logrank Test: Error Control under Continuous Monitoring with Unlimited Horizon" and Turner, Ly and Grunwald (2021) "Safe Tests and Always-Valid Confidence Intervals for contingency tables and beyond" for details on safe contingency table tests.

Functions to design and apply tests that are anytime valid. The functions can be used to design hypothesis tests in the prospective/randomised control trial setting or in the observational/retrospective setting. The resulting tests remain valid under both optional stopping and optional continuation.

E variables are tools for designing tests that keep their type-I error guarantees under flexible sampling scenarios such as optional stopping and continuation. We extend the recently developed E variables for two-sample tests to general null hypotheses and the corresponding anytime-valid confidence sequences. Using the 2x2 contingency table (Bernoulli) setting as a running example, we provide simple implementations of these confidence sequences for linear and odds-ratio based effect size.

Background: Developing predictive models for precision psychiatry is challenging because of unavailability of the necessary data: extracting useful information from existing electronic health record (EHR) data is not straightforward, and available clinical trial datasets are often not representative for heterogeneous patient groups. The aim of this study was constructing a natural language processing (NLP) pipeline that extracts variables for building predictive models from EHRs. We specifically tailor the pipeline for extracting information on outcomes of psychiatry treatment trajectories, applicable throughout the entire spectrum of mental health disorders (“transdiagnostic”). Methods: A qualitative study into beliefs of clinical staff on measuring treatment outcomes was conducted to construct a candidate list of variables to extract from the EHR. To investigate if the proposed variables are suitable for measuring treatment effects, resulting themes were compared to transdiagnostic outcome measures currently used in psychiatry research and compared to the HDRS (as a gold standard) through systematic review, resulting in an ideal set of variables. To extract these from EHR data, a semi-rule based NLP pipeline was constructed and tailored to the candidate variables using Prodigy. Classification accuracy and F1-scores were calculated and pipeline output was compared to HDRS scores using clinical notes from patients admitted in 2019 and 2020. Results: Analysis of 34 questionnaires answered by clinical staff resulted in four themes defining treatment outcomes: symptom reduction, general well-being, social functioning and personalization. Systematic review revealed 242 different transdiagnostic outcome measures, with the 36-item Short-Form Survey for quality of life (SF36) being used most consistently, showing substantial overlap with the themes from the qualitative study. Comparing SF36 to HDRS scores in 26 studies revealed moderate to good correlations (0.62—0.79) and

Functions to design and apply tests that are anytime valid. The functions can be used to design hypothesis tests in the prospective/randomised control trial setting or in the observational/retrospective setting. The resulting tests remain valid under both optional stopping and optional continuation. The current version includes safe t-tests and safe tests of two proportions. For details on the theory of safe tests, see Grunwald, de Heide and Koolen (2019) "Safe Testing" , for details on safe logrank tests see ter Schure, Perez-Ortiz, Ly and Grunwald (2020) "The Safe Logrank Test: Error Control under Continuous Monitoring with Unlimited Horizon" and Turner, Ly and Grunwald (2021) "Safe Tests and Always-Valid Confidence Intervals for contingency tables and beyond" for details on safe contingency table tests.

We present tests for conditional independence of discrete variables that can be applied in a sequential setting with Type-I error probability guarantee. Power of these tests can be improved by incorporating hypothesized effect size or by sharing information between strata. Both scenarios are illustrated through simulations.

We develop sequential A/B tests with strict Type-I error control under optional stopping. Our tests are based on E-variables, which recently have turned out be successful tools for anytime-valid inference. We introduce a general method for constructing E-variables that can be used for A/B testing in 2-sample streams. In contrast to earlier methods developed in the sequential testing literature, our approach is valid for both balanced and unbalanced experiments and allows for arbitrary, user-specified notions of effect size. The same method can be used to design anytime-valid confidence sequences to estimate effect sizes in data streams. With two Bernoulli streams as a running example, we illustrate the power of our A/B test and show that decisions can often be made earlier compared to classical methods, such as Fisher's exact test. We also illustrate the confidence sequences with two different notions of effect size: log odds ratio and difference in mean.

We develop E variables for testing whether two data streams come from the same source or not, and more generally, whether the difference between the sources is larger than some minimal effect size. These E variables lead to tests that remain safe, i.e. keep their Type-I error guarantees, under flexible sampling scenarios such as optional stopping and continuation. In special cases our E variables also have an optimal `growth' property under the alternative. We illustrate the generic construction through the special case of 2x2 contingency tables, where we also allow for the incorporation of different restrictions on a composite alternative. Comparison to p-value analysis in simulations and a real-world example show that E variables, through their flexibility, often allow for early stopping of data collection, thereby retaining similar power as classical methods.

We develop E-variables for testing whether two or more data streams come from the same source or not, and more generally, whether the difference between the sources is larger than some minimal effect size. These E-variables lead to exact, nonasymptotic tests that remain safe, i.e. keep their type-I error guarantees, under flexible sampling scenarios such as optional stopping and continuation. In special cases our E-variables also have an optimal 'growth' property under the alternative. While the construction is generic, we illustrate it through the special case of k x 2 contingency tables, where we also allow for the incorporation of different restrictions on a composite alternative. Comparison to p-value analysis in simulations and a real-world example show that E-variables, through their flexibility, often allow for early stopping of data collection, thereby retaining similar power as classical methods, while also retaining the option of extending or combining data afterwards.

We develop E variables for testing whether two data streams come from the same source or not, and more generally, whether the difference between the sources is larger than some minimal effect size. These E variables lead to tests that remain safe, i.e. keep their Type-I error guarantees, under flexible sampling scenarios such as optional stopping and continuation. In special cases our E variables also have an optimal `growth' property under the alternative. We illustrate the generic construction through the special case of 2x2 contingency tables, where we also allow for the incorporation of different restrictions on a composite alternative. Comparison to p-value analysis in simulations and a real-world example show that E variables, through their flexibility, often allow for early stopping of data collection, thereby retaining similar power as classical methods.

See the original psynlp package https://github.com/vmenger/psynlp for documentation, and some very nice examples and tutorials. The folder NLPpipeline contains the code for a pipeline with several consecutive filters for selecting sentences (in Dutch) containing phrases about four mental health outcome measures and change indicators. The code in this folder uses the modified psynlp package forked in this repository.