Search

Replication provides a confrontation of psychological theory, not only in experimental research, but also in model-based research. Goodness of fit (GOF) of the original model to the replication data is routinely provided as meaningful evidence of replication. We demonstrate, however, that GOF obscures important differences between the original and replication studies. As an alternative, we present Bayesian prior predictive similarity checking: a tool for rigorously evaluating the degree to which the data patterns and parameter estimates of a model replication study resemble those of the original study. We apply this method to original and replication data from the National Comorbidity Survey. Both data sets yielded excellent GOF, but the similarity checks often failed to support close or approximate empirical replication, especially when examining covariance patterns and indicator thresholds. We conclude with recommendations for applied research, including registered reports of model-based research, and provide extensive annotated R code to facilitate future applications of prior predictive similarity checking. [ABSTRACT FROM AUTHOR]

Deep Neural Networks (DNNs) often demonstrate remarkable performance when evaluated on the test dataset used during model creation. However, their ability to generalize effectively when deployed is crucial, especially in critical applications. One approach to assess the generalization capability of a DNN model is to evaluate its performance on replicated test datasets, which are created by closely following the same methodology and procedures used to generate the original test dataset. Our investigation focuses on the performance degradation of pre-trained DNN models in multi-class classification tasks when evaluated on these replicated datasets; this performance degradation has not been entirely explained by generalization shortcomings or dataset disparities. To address this, we introduce a new evaluation framework that leverages uncertainty estimates generated by the models studied. This framework is designed to isolate the impact of variations in the evaluated test datasets and assess DNNs based on the consistency of their confidence in their predictions. By employing this framework, we can determine whether an observed performance drop is primarily caused by model inadequacy or other factors. We applied our framework to analyze 564 pre-trained DNN models across the CIFAR-10 and ImageNet benchmarks, along with their replicated versions. Contrary to common assumptions about model inadequacy, our results indicate a substantial reduction in the performance gap between the original and replicated datasets when accounting for model uncertainty. This suggests a previously unrecognized adaptability of models to minor dataset variations. Our findings emphasize the importance of understanding dataset intricacies and adopting more nuanced evaluation methods when assessing DNN model performance. This research contributes to the development of more robust and reliable DNN models, especially in critical applications where generalization performance is of utmost importance. The code to reproduce our experiments will be available at . [ABSTRACT FROM AUTHOR]

The sluggish pace of new antibacterial drug development reflects a vulnerability in the face of the current severe threat posed by bacterial resistance. Microbial natural products (NPs), as a reservoir of immense chemical potential, have emerged as the most promising avenue for the discovery of next generation antibacterial agent. Directly accessing the antibacterial activity of potential products derived from biosynthetic gene clusters (BGCs) would significantly expedite the process. To tackle this issue, we propose a CSEL-BGC framework that integrates machine learning (ML) techniques. This framework involves the development of a novel cascade-stacking ensemble learning (CSEL) model and the establishment of a groundbreaking model evaluation system. Based on this framework, we predict 6,666 BGCs with antibacterial activity from 3,468 complete bacterial genomes and elucidate a biosynthetic evolutionary landscape to reveal their antibacterial potential. This provides crucial insights for interpretating the synthesis and secretion mechanisms of unknown NPs. [ABSTRACT FROM AUTHOR]

Background: Achieving complete immune reconstitution (CIR) in people living with human immunodeficiency virus (PLWH) following antiretroviral therapy (ART) is essential for preventing acquired immunodeficiency syndrome (AIDS) progression and improving survival. However, there is a paucity of robust prediction models for determining the likelihood of CIR in PLWH after ART. We aimed to develop and validate a CIR prediction model utilizing baseline data. Methods: Baseline data including demographic information, immunological profiles, and routine laboratory test results, were collected from PLWH in Yunnan, China. Baseline referred to the first recorded results after HIV diagnosis but before initiating ART, and these initial measurements served as the baseline data for analysis. The participants were divided into training and validation sets (7:3 ratio). To construct the model and accompanying nomogram, univariable and multivariable Cox regression analyses were performed. The model was evaluated using the C-index, time-dependent receiver operating characteristic (ROC) curves, calibration curves, and clinical decision curves to assess discrimination, calibration, and clinical applicability. Results: Five thousand four hundred eight PLWH were included, with a CIR of 38.52%. Cox regression analysis revealed various independent factors associated with CIR, including infection route, baseline CD4+T cell count, baseline CD4/CD8 ratio, interval from HIV diagnosis to ART initiation, and the level of PLT, Glu, Crea, HGB, ALT. A nomogram was formulated to predict the probability of achieving CIR at years 4, 5, and 6. The model demonstrated good performance, as evidenced by an AUC of 0.8 for both sets. Calibration curve analysis demonstrated a high level of agreement, and decision curve analysis revealed a significant positive yield. Conclusions: This study successfully developed a prediction model with robust performance. This model has considerable potential to aid clinicians in tailoring treatment strategies, which could enhance outcomes and quality of life for PLWH. [ABSTRACT FROM AUTHOR]

This paper introduces a new probabilistic composite model for the detection of zero-day exploits targeting the capabilities of existing anomaly detection systems in terms of accuracy, computational time, and adaptability. To address the issues mentioned above, the proposed framework consisted of three novel elements. The first key innovations are the introduction of "Adaptive WavePCA-Autoencoder (AWPA)" for pre-processing stage which address the denoising and dimensionality reduction, and contributes to the general dependability and accuracy of zero-day exploit detection. Additionally, a novel "Meta-Attention Transformer Autoencoder (MATA)" for enhancing feature extraction which address the subtlety issue, and improves the model's ability and flexibility to detect new security threats, and a novel "Genetic Mongoose-Chameleon Optimization (GMCO)" was introduced for effective feature selection in the case of addressing the efficiency challenges. Furthermore, a novel "Adaptive Hybrid Exploit Detection Network (AHEDNet)" was introduced which address the dynamic ensemble adaptation issue where the accuracy of anomaly detection is very high with low false positives. The experimental results show the proposed model outperforms the other models of dataset 1 in accuracy of 0.988086 and 0.990469, precision of 0.987976 and 0.990628, recall of 0.988298 and 0.990435, with the lowest Hamming Loss of 0.011914 and 0.009531, also, the proposed model outperforms the other models of dataset 2 in accuracy of 0.9819 and 0.9919, precision of 0.9868 and 0.9968, recall of 0.9813 and 0.9923, with the lowest Hamming Loss of 0.0209 and 0.0109, thus the proposed model outperformed the other models in detecting zero-day exploits. [ABSTRACT FROM AUTHOR]

Testing is an important aspect of software development. Automation testing is now widely used to achieve better and more efficient results. Various automation testing frameworks are available in the market. However, one of the major challenges is determining which automation testing framework is suitable for testing. This study proposes an evaluation model for evaluating web automation testing frameworks based on seven performance efficiency factors to address this issue. The model evaluates five types of transactions commonly used on the web; CRUD, Get Massive Data, search, file upload, and file download. In addition, the tested frameworks are categorized as good, medium, and low. To measure the success of the research, expert weighting was also used. Based on the results obtained for all types of transactions, almost all classifications between the experimental results and weighting were in the same class. Although the model was found to be effective with a 100% accuracy rate, it had an accuracy rate of 80% for upload transactions. The outcomes of this study serve as a valuable reference for choosing suitable software for both tested frameworks and other software applications. In future studies focus on narrowing the selection based on not only performance but also functionality and ease of use. [ABSTRACT FROM AUTHOR]

This study aimed to compare and evaluate the prediction accuracy and risk of bias (ROB) of post-traumatic stress disorder (PTSD) predictive models. We conducted a systematic review and random-effect meta-analysis summarizing predictive model development and validation studies using machine learning in diverse samples to predict PTSD. Model performances were pooled using the area under the curve (AUC) with a 95% confidence interval (CI). Heterogeneity in each meta-analysis was measured using I2. The risk of bias in each study was appraised using the PROBAST tool. 48% of the 23 included studies had a high ROB, and the remaining had unclear. Tree-based models were the primarily used algorithms and showed promising results in predicting PTSD outcomes for various groups, as indicated by their pooled AUCs: military incidents (0.745), sexual or physical trauma (0.861), natural disasters (0.771), medical trauma (0.808), firefighters (0.96), and alcohol-related stress (0.935). However, the applicability of these findings is limited due to several factors, such as significant variability among the studies, high and unclear risks of bias, and a shortage of models that maintain accuracy when tested in new settings. Researchers should follow the reporting standards for AI/ML and adhere to the PROBAST guidelines. It is also essential to conduct external validations of these models to ensure they are practical and relevant in real-world settings. [ABSTRACT FROM AUTHOR]

Multi-label ranking, which returns multiple top-ranked labels for each instance, has a wide range of applications for visual tasks. Due to its complicated setting, prior arts have proposed various measures to evaluate model performances. However, both theoretical analysis and empirical observations show that a model might perform inconsistently on different measures. To bridge this gap, this paper proposes a novel measure named Top-K Pairwise Ranking (TKPR), and a series of analyses show that TKPR is compatible with existing ranking-based measures. In light of this, we further establish an empirical surrogate risk minimization framework for TKPR. On one hand, the proposed framework enjoys convex surrogate losses with the theoretical support of Fisher consistency. On the other hand, we establish a sharp generalization bound for the proposed framework based on a novel technique named data-dependent contraction. Finally, empirical results on benchmark datasets validate the effectiveness of the proposed framework. [ABSTRACT FROM AUTHOR]

Understanding how ecosystem change influences fishery resources through trophic pathways is a key tenet of ecosystem‐based fishery management. System models (SM), which use numerical modeling to describe physical and biological processes, can advance inclusion of ecosystem and prey information in fisheries management; however, incorporating SMs in management requires evaluation against empirical data. The Bering Ecosystem Study Nutrient‐Phytoplankton‐Zooplankton (BESTNPZ) model is an SM (originally created by the Bering Ecosystem Study, which initiated in 2006 and was expanded by Kearney et al.) includes zooplankton biomass hindcasts for the Bering Sea. In the Bering Sea, zooplankton are an important prey item for fishery species, yet the zooplankton component of this SM has not been validated against empirical data. We compared empirical zooplankton data to BESTNPZ hindcast estimates for three zooplankton functional groups and found that the two sources of information are on different absolute scales. We found high correlation between relative seasonal biomass trends estimated by BESTNPZ and empirical data for large off‐shelf copepods (Neocalanus spp.) and low correlations for large on‐shelf copepods and small copepods (Calanus spp. and Pseudocalanus spp., respectively). To address these discrepancies, we constructed hybrid species distribution models (H‐SDM), which predict zooplankton biomass using the BESTNPZ hindcast and environmental covariates. We found that H‐SDMs offered marginal improvements over correlative species distribution models (C‐SDMs) relying solely on empirical data for spatial extrapolation and little improvement for most functional groups when forecasting short‐term temporal zooplankton biomass trends. Overall, we suggest that interpretation of current BESTNPZ hindcasts should be tempered by our understanding of key mismatches in absolute scale, seasonality, and annual indices between BESTNPZ and empirical data. [ABSTRACT FROM AUTHOR]

Background: Wildfire simulation models are used to derive maps of burn probability (BP) based on fuels, weather, topography and ignition locations, and BP maps are key components of wildfire risk assessments. Aims: Few studies have compared BP maps with real-world fires to evaluate their suitability for near-future risk assessment. Here, we evaluated a BP map for the conterminous US based on the large fire simulation model FSim. Methods: We compared BP with observed wildfires from 2016 to 2022 across 128 regions representing similar fire regimes ('pyromes'). We evaluated the distribution of burned areas across BP values, and compared burned area distributions among fire size classes. Key results: Across all pyromes, mean BP was moderately correlated with observed burned area. An average of 71% of burned area occurred in higher-BP classes, vs 79% expected. BP underpredicted burned area in the Mountain West, especially for extremely large fires. Conclusions: The FSim BP map was useful for estimating subsequent wildfire hazard, but may have underestimated burned areas where input data did not reflect recent climate change, vegetation change or human ignition patterns. Implications: Our evaluations indicate that caution is needed when relying on simulation-based BP maps to inform management decisions. Our results also highlight potential opportunities to improve model estimates. An FSim burn probability (BP) map for the conterminous US was compared with observed wildfires from 2016 to 2022 in order to evaluate the suitability of wildfire simulation models for quantifying near-future wildfire risk. Our results highlight where burn probability models may be improved by incorporating dynamic weather and human influences. [ABSTRACT FROM AUTHOR]

Abstract In vitro rumen gas production experiment was conducted with 57 kinds of feedstuff, which were categorized into energy feed, protein feed, and roughage, collected within China. Eight mathematical models were employed to describe the kinetics of in vitro rumen gas production. The results found that for energy feeds, protein feeds, and roughages, respectively, Michaelis–Menten (MM) or Logistic-Exponential with lag (LEL), MM, and Mitscherlich (MIT) exhibited the highest or shown no significant difference compared to the highest coefficient of determination (R2) (P

AIM: To analyze the modeling characteristics and validation indexes of diabetic retinopathy model, analyze the shortcomings of the present animal experimental modeling, and provide a reference basis for the establishment of the standardization of the diabetic retinopathy model.METHODS: Literatures related to animal experiments on diabetic retinopathy were searched through the databases of CNKI, Wanfang, VIP and PubMed, and the experimental animal species, grade, gender, age, modeling method, modeling period, validation indexes, and other indexes were summarized and analyzed.RESULTS: The 275 papers that met the criteria were included. The animal models of diabetic retinopathy were mainly SD rats and Wistar rats, the sex of the experimental animals was mainly male, and the animal breeds were mostly of the SPF class. The age of most of the animals used was in the range of 6-8 weeks old; the modeling was based on those who established the type 1 diabetes model, mainly using STZ as the induction model. While the type 2 diabetes model was based on the high-fat, high-sugar diet combined with STZ. The modeling criteria were verified by detecting retinal morphology and structure, retinal vascularization, retinal function, and retinal cell apoptosis. In addition, the model was evaluated by detecting abnormal vascular proliferation, oxidative stress indicators, and inflammatory factor levels in retinal tissues, as well as abnormal vascular proliferation and inflammatory factor levels in aqueous humour, vitreous humor, and blood serum.CONCLUSION: Although the animal model of diabetic retinopathy has become a hotspot, the existing review is not comprehensive. Therefore, we summarized and analyzed the elements of the animal model through literature collation, including its characteristics and limitations, and providing methodological references for the establishment of the model, with a view to laying a solid foundation for the subsequent clinical and basic research of traditional Chinese medicine.

Methane (CH4) emissions from urban sewer systems represent a significant contributor to greenhouse gases, driven by anaerobic decomposition processes. This review elucidates the mechanisms underlying CH4 production in sewers, which are influenced by environmental factors such as the COD/SO42− ratio, temperature, dissolved oxygen, pH, flow rate, and hydraulic retention time. We critically evaluated the effectiveness of empirical, mechanistic, and machine learning (ML) models in predicting CH4 emissions, highlighting the limitations of each. This review further examines control strategies, including oxygen injection, iron salt dosing, and nitrate application, emphasizing the importance of balancing CH4 reduction with the operational efficiency of wastewater treatment plants (WWTPs). An integrated approach combining mechanistic and data-driven models is advocated to enhance prediction accuracy and optimize CH4 management across urban sewer systems. [ABSTRACT FROM AUTHOR]

Transferability of species distribution models (SDMs) is vital for guiding biological conversation in the changing marine ecosystems, whereas most SDMs have remained insufficiently tested in terms of transferences. Particularly, the roles of biotic interactions in shaping ecological niches had less been considered when predictive models are transferred into a new circumstance. Therefore, using silver croaker (Pennahia argentata) and slender lizardfish (Saurida elongata), two keystone predators in the coastal systems as examples, this study examined how the consideration of abiotic and biotic factors as well as spatial autocorrelation would improve the transferability of prevalent SDMs. We collected the data of species abundance and hydrological variables from coastal waters of Shandong Peninsula, China, and developed a framework to evaluate the interpolation performance and spatial transferability of SDMs with respect to different modelling algorithms and combinations of covariates. The covariates tested included (1) abiotic environmental variables only; (2) biotic variables of prey availability only; (3) both abiotic and biotic variables; and (4) spatial autocorrelation in addition to other variables. Generally, our results showed that including abiotic and biotic variables simultaneously could improve the model transferability, whereas biotic models exhibited greater uncertainty and overestimation in spatial transferences when delineating suitable habitats. Besides, models accounting for spatial autocorrelation yielded more accurate forecasts for slender lizardfish's abundance, and were less limited by geographic distance in transference. Regarding the model algorithms, GLM slightly outperformed GAM and RF overall, while RF showed more robust spatial extrapolations in many cases. Our transferability evaluations offer the opportunity to extend the use of biotic signals and spatial autocorrelation to improve the forecasts of habitat suitability, which would be informative for guiding marine conservation actions. [ABSTRACT FROM AUTHOR]

AIM: To establish a mouse model of pulmonary fibrosis induced by multiple intratracheal instillations of bleomycin (BLM). METHODS: C57BL/6J mice were randomly divided into five groups: control group (n=5), multiple high-dose BLM(BLM-MH) group(n=10), multiple medium-dose BLM(BLM-MM) group(n=8), multiple lowdose BLM(BLM-ML) group(n=7), and single medium-dose BLM(BLM-SM) group(n=6). The pulmonary fibrosis model was induced by single or multiple intratracheal instillations of BLM. Survival curves were plotted at day 56, and lung tissue was collected for lung coefficient calculation. Pathological changes in lung tissue were assessed using hematoxylin-eosin(H&E) staining and Masson staining. Indicators of lung fibrosis, such as hydroxyproline(HYP), were measured. Differentially expressed genes in patients with IPF were screened using the GEO database, and kyoto encyclopedia of genes and genomes (KEGG) pathway enrichment analysis was performed to identify pathways associated with IPF. Western blot was used to detect changes in these pathways in the lung tissues of the model mice. RESULTS: The survival rates were 50% in the BLM-MH group and 87. 5% in the BLM-MM group, with no deaths in other groups. Compared with the control group, the BLM-MH and BLM-MM groups showed significantly increased lung coefficients (P<0. 05), lung tissue damage, inflammation levels, degree of pulmonary fibrosis(P<0. 05 or P<0. 01), and HYP content. The BLM-MH and BLMMM groups, compared with the BLM-SM group, showed significantly elevated levels of inflammation, fibrosis, and HYP content (P<0. 05 or P<0. 01). GEO database and KEGG enrichment analysis revealed that the extracellular matrix-receptor interaction pathway (ECM-RIP) may be involved in IPF development. The expression levels of ECM-RIP pathway-related proteins, such as phosphorylated focal adhesion kinase (p-FAK) and phosphorylated Src protein (p-Src), were increased in the lung tissues of the model mice compared with the control group. Compared with the BLM-SM group, the BLM-MH group exhibited significant increases in the protein levels of p-FAK and p-Src in the lung tissue (P<0. 05). CONCLUSION: The murine model of pulmonary fibrosis established through repeated intratracheal instillations of BLM effectively mimics the pathological characteristics of the disease, providing a valuable experimental model for the investigation of idiopathic pulmonary fibrosis pathogenesis and for the development of therapeutic agents. [ABSTRACT FROM AUTHOR]

Wildfires pose a serious threat to ecosystems and human safety, and with the backdrop of global climate change, the prediction of forest fires has become increasingly important. Traditional machine learning methods face challenges in forest fire prediction, such as difficulty identifying feature parameters, manual intervention in model selection, and hyperparameter tuning, which affect prediction accuracy and efficiency. This study proposes an analytical framework for forest fire prediction based on Automated Machine Learning (AutoML) technology to address the challenges traditional machine learning methods face in forest fire prediction. We collected meteorological, topographical, and vegetation data from Guangxi Province, with meteorological data covering 1994 to 2023, providing comprehensive background information for our prediction model. Using the prediction model, which was constructed with the AutoGluon framework, the experimental results indicate that models under the AutoGluon framework (e.g., KNeighborsDist classifier) significantly outperform traditional machine learning models in terms of accuracy, precision, recall, and F1-Score, with the highest accuracy rate reaching 0.960. Model error analysis shows that models under the AutoGluon framework perform better in error control. This study provides an efficient and accurate method for forest fire prediction, which is of great significance for decision-making in forest fire management and for protecting forest resources and ecological security. [ABSTRACT FROM AUTHOR]

Background: Software defect prediction (SDP) is a topic actively researched in the software engineering community. Within‐project defect prediction (WPDP) involves using labeled modules from previous versions of the same project to train classifiers. Over time, many defect prediction models have been evaluated under the WPDP scenario. Problem: Data duplication poses a significant challenge in current WPDP evaluation procedures. Unchanged modules, characterized by identical executable source code, are frequently present in both target and source versions during experimentation. However, it is still unclear how and to what extent the presence of unchanged modules affects the performance assessment of WPDP models and the comparison of multiple WPDP models. Method: In this paper, we provide a method to detect and remove unchanged modules from defect datasets and unveil the impact of data duplication in WPDP on model evaluation. Results: The experiments conducted on 481 target versions from 62 projects provide evidence that data duplication significantly affects the reported performance values of individual learners in WPDP. However, when ranking multiple WPDP models based on prediction performance, the impact of removing unchanged instances is not substantial. Nevertheless, it is important to note that removing unchanged instances does have a slight influence on the selection of models with better generalization. Conclusion: We recommend that future WPDP studies take into consideration the removal of unchanged modules from target versions when evaluating the performance of their models. This practice will enhance the reliability and validity of the results obtained in WPDP research, leading to improved understanding and advancements in defect prediction models. [ABSTRACT FROM AUTHOR]

Large language models (LLMs) have shown their capabilities in numerical and logical reasoning, yet their capabilities in higher-order cognitive tasks, particularly causal reasoning, remain less explored. Current research on LLMs in causal reasoning has focused primarily on tasks such as identifying simple cause-effect relationships, answering basic "what-if" questions, and generating plausible causal explanations. However, these models often struggle with complex causal structures, confounding variables, and distinguishing correlation from causation. This work addresses these limitations by systematically evaluating LLMs' causal reasoning abilities across three representative scenarios, namely analyzing causation from effects, tracing effects back to causes, and assessing the impact of interventions on causal relationships. These scenarios are designed to challenge LLMs beyond simple associative reasoning and test their ability to handle more nuanced causal problems. For each scenario, we construct four paradigms and employ three types of prompt scheme, namely zero-shot prompting, few-shot prompting, and Chain-of-Thought (CoT) prompting in a set of 36 test cases. Our findings reveal that most LLMs encounter challenges in causal cognition across all prompt schemes, which underscore the need to enhance the cognitive reasoning capabilities of LLMs to better support complex causal reasoning tasks. By identifying these limitations, our study contributes to guiding future research and development efforts in improving LLMs' higher-order reasoning abilities. [ABSTRACT FROM AUTHOR]

Chlorophyll-a (Chl-a) serves as a crucial indicator of water quality, making the precise monitoring of its concentration essential for aquatic environment ecosystem protection. Water color retrieval technology has gained prominence in monitoring spatiotemporal variations in water quality. This study evaluated inversion models for Chl-a estimation in urban water bodies using ground-based hyperspectral data in Nanjing, China. The results indicate that the normalizing of water-leaving reflectance significantly enhances the correlation between water-leaving reflectance and measured Chl-a concentration. However, due to the complexity of urban water bodies and the diversity of interfering components, the three ratio algorithms of OC2V4, OC4V4, and TChla using blue–green bands yielded suboptimal inversion results. In contrast, the normalized fluorescence line height (NFH) algorithm exhibited a robust performance, yielding an R2 of 0.70. Furthermore, the overall performance of the near-infrared–Red (NIR-red)-band algorithms showed a commendable overall performance (R2 > 0.60), and the best four-band algorithm, 4BDA, achieved an R2 of 0.72. Other index algorithms, such as the Yang index and the normalized difference Chl-a index (NDCI), also performed well (R2 = 0.61). Notably, the classification of Chl-a concentrations did not significantly enhance the inversion accuracy of the empirical and semi-analytical models. Only the NFH algorithm using the fluorescence band greatly improved the inversion accuracy for low Chl-a concentrations (R2 = 0.75), likely due to the influence of Chl-a and other substances on fluorescence peak positioning and height. Ultimately, the NFH model is identified as the optimal approach for Chl-a inversion across varying Chl-a concentrations in urban water bodies. This study provides critical technical support for the protection of aquatic environments and the management of urban water resources. [ABSTRACT FROM AUTHOR]