Search

Clinical Practice Guidelines (CPGs) gather latest evidence-based results to guide and support clinicians over the decision-making process to provide best care. Nevertheless, clinical cases may be subject to some biases (understood as non-compliance with CPGs) that can lead to adapt care delivery. In this work an experience-based decision support leaning on the structuration of the Decisional Event concept for tracking and storing each clinical decision is presented. Moreover, a visual analytics tool is provided in order to facilitate the visualization of biases from guideline-based decision support and the identification and inclusion of real-world evidence into the reasoning process by augmenting the knowledge formalized in the implemented guidelines.

Potentially inappropriate prescribing of direct oral anticoagulants is frequent with the most common errors being dosage, administration, and duration of therapy. We developed RecosDoc-MTeV, a documentary-based clinical decision support system (CDSS) for the management of direct oral anticoagulant prescription to prevent and treat venous thromboembolism. Simultaneously, the network of Parisian public hospitals (AP-HP, France) developed narrative clinical practice guidelines (CPGs) and a companion smartphone application to enhance medication and patient safety related to direct oral anticoagulant prescription. To assess the effectiveness of these CDS tools, we performed a retrospective review of 274 random patients hospitalized in 2017, which were either at risk of venous thromboembolism or actually treated for the disease. Consistency between the two CDS tools was measured at 96.7%. Administered treatments were compliant in 67.2% and 72.3% of the cases, with AP-HP CPGs and RecosDoc-MTeV, respectively. These results support that implementing CDSSs for the prescription of direct oral anticoagulants may ensure safe prescribing of high-risk medications.

To summarize recent research and select the best papers published in 2018 in the field of computerized clinical decision support for the Decision Support section of the International Medical Informatics Association (IMIA) yearbook. A literature review was performed by searching two bibliographic databases for papers referring to clinical decision support systems (CDSSs). The aim was to identify a list of candidate best papers from the retrieved bibliographic records, which were then peer-reviewed by external reviewers. A consensus meeting of the IMIA editorial team finally selected the best papers on the basis of all reviews and the section editors' evaluation. Among 1,148 retrieved articles, 15 best paper candidates were selected, the review of which resulted in the selection of four best papers. The first paper introduces a deep learning model for estimating short-term life expectancy (>3 months) of metastatic cancer patients by analyzing free-text clinical notes in electronic medical records, while maintaining the temporal visit sequence. The second paper takes note that CDSSs become routinely integrated in health information systems and compares statistical anomaly detection models to identify CDSS malfunctions which, if remain unnoticed, may have a negative impact on care delivery. The third paper fairly reports on lessons learnt from the development of an oncology CDSS using artificial intelligence techniques and from its assessment in a large US cancer center. The fourth paper implements a preference learning methodology for detecting inconsistencies in clinical practice guidelines and illustrates the applicability of the proposed methodology to antibiotherapy. Three of the four best papers rely on data-driven methods, and one builds on a knowledge-based approach. While there is currently a trend for data-driven decision support, the promising results of such approaches still need to be confirmed by the adoption of these systems and their routine use.

DESIREE is a European-funded project to improve the management of primary breast cancer. We have developed three decision support systems (DSSs), a guideline-based, an experience-based, and a case-based DSSs, resp. GL-DSS, EXP-DSS, and CB-DSS, that operate simultaneously to offer an enriched multi-modal decision support to clinicians. A breast cancer knowledge model has been built to describe within a common ontology the data model and the termino-ontological knowledge used for representing breast cancer patient cases. It allows for rule-based and subsumption-based reasoning in the GL-DSS to provide best patient-centered reconciled care plans. It also allows for using semantic similarity in the retrieval algorithm implemented in the CB-DSS. Rainbow boxes are used to display patient cases similar to a given query patient. This innovative visualization technique translates the question of deciding the most appropriate treatment into a question of deciding the colour dominance among boxes.

Technologies such as decision support systems are expected to help clinicians implement clinical practice guidelines (CPGs) with the aim of decreasing practice variations and improving clinical outcomes. However, if CPGs provide recommendations to improve patient care, they may fail to take into account actual clinical outcomes associated to the recommended treatment, such as adverse events or secondary effects. In this paper, we present a novel experience-based decision support approach applied to the management of breast cancer, the most commonly diagnosed cancer among women worldwide. Capitalizing on the clinical know-how of physicians and the modeling of patient's outcomes and toxicities in a computer interpretable way, we are able to discover new knowledge that helps improving patient-centered clinical care. This work is conducted within the EU Horizon 2020 project DESIREE.

Breast cancer is the most common cancer among women. DESIREE is a European project which aims at developing web-based services for the management of primary breast cancer by multidisciplinary breast units (BUs). We describe the guideline-based decision support system (GL-DSS) of the project. Various breast cancer clinical practice guidelines (CPGs) have been selected to be concurrently applied to provide state-of-the-art patient-specific recommendations. The aim is to reconcile CPG recommendations with the objective of complementarity to enlarge the number of clinical situations covered by the GL-DSS. Input and output data exchange with the GL-DSS is performed using FHIR. We used a knowledge model of the domain as an ontology on which relies the reasoning process performed by rules that encode the selected CPGs. Semantic web tools were used, notably the Euler/EYE inference engine, to implement the GL-DSS. “Rainbow boxes” are a synthetic tabular display used to visualize the inferred recommendations.

Clinical decision support systems (CDSSs) have proven to potentially improve the compliance of physician decisions with clinical practice guidelines (CPGs). However, actual patients suffer from multiple conditions and CPGs that are usually single-disease-focused provide disease-specific recommendations with no support on how to manage adverse interactions between the recommended treatments. We have developed GO-DSS, a CDSS that implements an ontological reasoning process to perform CPG reconciliation. GO-DSS is applied to the concurrent management of hypertension, type 2 diabetes, and dyslipidemia. We proposed an innovative graphical interface to display medication recommendations as "therapeutic circles". A qualitative evaluation of the system and of this graphical layout has been performed on simulated patient cases by a sample of 12 users with various backgrounds (think aloud method). The resulting usability of the system is highly appreciated with a mean rating of 90.7% according to the standardized System Usability Scale.

International audience; Scoring sleep stages can be considered as a classification problem. Once the whole recording segmented into 30-seconds epochs, features, extracted from raw signals, are typically injected into machine learning algorithms in order to build a model able to assign a sleep stage, trying to mimic what experts have done on the training set. Such approaches ignore the advances in sleep medicine, in which guidelines have been published by the AASM, providing definitions and rules that should be followed to score sleep stages. In addition, these approaches are not able to solve conflict situations, in which criteria of different sleep stages are met. This work proposes a novel approach based on AASM guidelines. Rules are formalized integrating, for some of them, preferences allowing to support decision in conflict situations. Applied to a doubtful epoch, our approach has taken the appropriate decision.

International audience; Polysomnography is the gold standard test for sleep disorders among which the Sleep Apnea Syndrome (SAS) is considered a public health issue because of the increase of the cardio-and cerebro-vascular risk it is associated with. However, the reliability of this test is questioned since sleep scoring is a time-consuming task performed by medical experts with a high inter-and intra-scorers variability, and because data are collected from 15 sensors distributed over a patient's body surface area, using a wired connection which may be a source of artefacts for the patient's sleep. We have used symbolic fusion to support the automated diagnosis of SAS on the basis of the international guidelines of the AASM for the scoring of sleep events. On a sample of 70 patients, and for the Apnea-Hypopnea Index, symbolic fusion performed at the level of sleep experts (97.1% of agreement). The next step is to confirm these preliminary results and move forward to a smart wireless polysomnograph.

Published as textual documents, clinical practice guidelines (CPGs) didn't demonstrate to impact physician practices when disseminated in their original format. However, when computerized and embedded in clinical decision support systems, they appeared to be more effective. In order to ease the translation from textual to computerized CPGs, we have elaborated a fine-grained knowledge model of CPGs (FGKM) to be used when authoring CPGs. The work has been conducted on VIDALRecos® CPGs. The building of the model has followed a bottom-up iterative process starting with 15 different CPGs. The first version of the FGKM has been assessed on two new complex CPGs, and was enriched by comparison with the Guideline Elements Model (GEM). The final version of the FGKM has been tested on the 2014 Hypertension CPGs. We compared the rules automatically derived from FGKM instances to those manually extracted from textual CPGs for decision support. Results showed that difficulties such as text normalization have to be solved. The FGKM is intended to be used upstream of the process of CPGs authoring in order to ease the implementation and the update of both textual and computerized CPGs.

Diagnostic imaging requisition (DIR) content is legally constrained for care quality and patient safety concerns. A French national indicator, based on administrative and clinical data, has been introduced to monitor nationwide the conformity of such documents (CDIR). The purpose of this study was to assess the effect on CDIR of the deployment of the ORBIS™ electronic medical record at the Tenon hospital (Paris, France). A before-after study has been carried out. A significant increase of CDIR, from 37.0% (n=676) to 49.1% (n=800), was observed (p < 10⁻⁵). Conformity of administrative criteria improved, but there was no statistical difference of clinical criteria conformity, despite the improvement of clinical history documentation (100%). Up to five different paper-based requisition forms were used by clinical departments in the before period. In the after period, only 27.1% of requisitions were ORBIS-edited with a CDIR of 66.8% (n=217). In both periods, CDIR was correlated to the level of standardization of the forms.

To summarize recent research and propose a selection of best papers published in 2014 in the field of computerized clinical decision support for the Decision Support section of the IMIA yearbook. A literature review was performed by searching two bibliographic databases for papers related to clinical decision support systems (CDSSs) and computerized provider order entry systems in order to select a list of candidate best papers to be then peer-reviewed by external reviewers. A consensus meeting between the two section editors and the editorial team was finally organized to conclude on the selection of best papers. Among the 1,254 returned papers published in 2014, the full review process selected four best papers. The first one is an experimental contribution to a better understanding of unintended uses of CDSSs. The second paper describes the effective use of previously collected data to tailor and adapt a CDSS. The third paper presents an innovative application that uses pharmacogenomic information to support personalized medicine. The fourth paper reports on the long-term effect of the routine use of a CDSS for antibiotic therapy. As health information technologies spread more and more meaningfully, CDSSs are improving to answer users' needs more accurately. The exploitation of previously collected data and the use of genomic data for decision support has started to materialize. However, more work is still needed to address issues related to the correct usage of such technologies, and to assess their effective impact in the long term.

Decision support systems, as means of disseminating clinical practice guidelines, are powerful software that may lead to an improvement of medical practices. However, they are not always efficient and may suffer from limitations among which are lack of flexibility and weaknesses in the integration of several clinical practice guidelines (CPGs) for the management of patients with multiple chronic disorders. We propose a framework based on an ontological modeling of CPG contents as rules. The ontology provides the required flexibility to adapt patient data and enable the provision of appropriate recommendations expressed at various levels of abstraction. To solve decisional conflicts that occur when combining multiple sources of recommendations, we proposed a method based on the subsumption graph of the patient profiles corresponding to the rules. A prototype CDSS implementing this approach has been developed. Results are given on a clinical case to illustrate the assets of ontological reasoning in increasing the number of issued recommendations and thereby the reliability of decision support.

Every year, numerous clinical practice guidelines (CPGs) are published on a same topic. They may be conflicting, thus infringing clinicians' confidence in adhering to them. In order to build a clinical decision support system to assist GPs in the management of hypertension, we have considered three recent CPGs written in French. We developed a methodological framework to evaluate how consistent the three CPGs were. After a manual extraction of recommendation rules, all patient profiles covered by the CPGs have been identified. Then, ontological modeling and reasoning were used to build a subsumption graph of all profiles. This graph allows the retrieval of recommendations that could be conflicting. Results show that if rules are different in the three CPGs according to a document-based approach, many profiles are related through subsumption, and no critical inconsistencies were discovered when implementing an ontological modeling.

The potential of health information technology is hampered by new types of errors which impact is not totally assessed. OncoDoc2 is a decision support system designed to support treatment decisions of multidisciplinary meetings (MDMs) for breast cancer patients. We evaluated how the way the system was used had an impact on MDM decision compliance with clinical practice guidelines. We distinguished "correct navigations" (N+), "incorrect navigations" (N-), and "missing navigations" (N0), according to the quality of data entry when using OncoDoc2. We collected 557 MDM decisions from three hospitals of Paris area (France) where OncoDoc2 was routinely used. We observed 33.9% N+, 36.8% N-, and 29.3% N0. The compliance rate was significantly different according to the quality of navigations, 94.2%, 80.0%, and 90.2% for N+, N-, and N0 respectively. Surprinsingly, it was better not to use the system (N0) than to use it improperly (N-).

To summarize excellent research and to select best papers published in 2012 in the field of computer-based decision support in healthcare. A bibliographic search focused on clinical decision support systems (CDSSs) and computer provider order entry was performed, followed by a double-blind literature review. The review process yielded six papers, illustrating various aspects of clinical decision support. The first paper is a systematic review of CDSS intervention trials in real settings, and considers different types of possible outcomes. It emphasizes the heterogeneity of studies and confirms that CDSSs can improve process measures but that evidence lacks for other types of outcomes, especially clinical or economic. Four other papers tackle the safety of drug prescribing and show that CDSSs can be efficient in reducing prescription errors. The sixth paper exemplifies the growing role of ontological resources which can be used for several applications including decision support. CDSS research has to be continuously developed and assessed. The wide variety of systems and of interventions limits the understanding of factors of success of CDSS implementations. A standardization in the characterization of CDSSs and of intervention trial reporting will help to overcome this obstacle.

By providing patient-specific advice, clinical decision support systems (CDSSs) are expected to promote the implementation of clinical practice guidelines (CPGs) to improve the quality of care. However, produced as texts, often incomplete and ambiguous, CPGs are difficult to translate into the formal knowledge bases (KBs) of CDSSs. The French National Authority for Health (HAS) decided to update CPGs on the management of type 2 diabetes. This work illustrates the simultaneous development of the text and its formal counterpart in a CDSS named RecosDiab. CPGs were elaborated by a working group according to the guideline development methodology. Textual recommendations were graded, either as evidence-based when evidence existed or as consensus-based when acknowledge by the working group. Knowledge modeling was performed following the steps of de-abstraction, disambiguation, and verification of completeness. This last step generated clinical situations not explicitly mentioned in the text and were graded as expert-based. The resulting KB provides therapeutic advice for 805 clinical situations, among which 2 are graded as evidence-based, 37 are consensus-based, and 766 are expert-based. However, because of the amount of expert-based propositions, the HAS did not endorse the system.

Assessing the conformity of a physician's prescription to a given recommended prescription is not obvious since both prescriptions are expressed at different levels of abstraction and may concern only a subpart of the whole order. Recent formalisms (OWL2) and tools (reasoners) from the semantic web technologies are becoming available to represent defined concepts and to handle classification services. We propose a generic framework based on such technologies, using available standardized drug resources, to compute the compliance of a given drug order to a recommended prescription, such that the subsumption relationship yields the conformity relationship between the order and the recommendation. The ATC drug classification has been used as a local ontology. The method has been successfully implemented for arterial hypertension management for which we had a sample of antihypertensive orders. However, supplemental standardized drug knowledge is needed to correctly compare drug orders to recommended orders.

International audience; Clinical decision support systems have been developed to help physicians to take clinical guidelines into account during consultations. The ASTI critiquing module is one such systems; it provides the physician with automatic criticisms when a drug prescription does not follow the guidelines. It was initially developed for hypertension and type 2 diabetes, but is designed to be generic enough for application to all chronic diseases. We present here the results of usability and satisfaction evaluations for the ASTI critiquing module, obtained with GPs for a newly implemented guideline concerning dyslipaemia, and we discuss the lessons learnt and the difficulties encountered when building a generic DSS for critiquing physicians' prescriptions.

In 2002, Haynes et al. founded a prescriptive model of evidence-based medicine based on the patient’s clinical state, her preferences, and research evidence, clinical expertise synthesizing the other three components. Revisiting this model of medical decision making, we propose a descriptive model introducing clinicians’ preferences and formalize four reasons of non-compliance with clinical practice guidelines (CPGs). The approach has been applied to breast cancer management decisions taken by multidisciplinary staff meetings (MSMs) at the Tenon hospital, Paris, France, while using a clinical decision support system (CDSS): OncoDoc2. 1,889 MSM decisions have been recorded [February 2007–October 2009]. The compliance rate with CPGs was measured at 91.0%. Non-compliant decisions are mainly “MSM choices” (39.1%) and “particular cases” (34.9%). “Practice evolution” and “patient choices” are less frequent (12.4% and 11.2%). Even with a CDSS, a 100% compliance rate cannot be reached because particular cases fall outside CPGs and borderline cases need to be interpreted by clinicians.

Clinical decision support systems (CDSSs) have the potential to increase guideline adherence, but factors of success are not yet understood. ASTI guiding mode (ASTI-GM) is an on-demand guideline-based CDSS where the user navigates in a knowledge base to get the best treatment for a given patient. We conducted a web-based evaluation of ASTI-GM, carried out as a before-after study, where general practitioners (GPs) were asked to solve 5 clinical cases, first without ASTI-GM, then using the system. Of the 136 GPs that resolved the case on the management of hypertension, compliance with best practices increased from 69.1% to 80.9% with ASTI-GM. When the navigation matched the set of patient parameters described in the clinical case, the increase was even higher and reached 92.9%. E-iatrogenesis has been measured at 19.1%, with 5.1% of commission errors, 8.1% of negative reactance, and 5.9% of neutral reactance. Role of physicians' reactance in noncompliance with guideline-based CDSSs should be further investigated.

Clinical decision support systems (CDSSs) have the potential to increase guideline adherence, but factors of success are not well understood. ASTI-GM is an on demand guideline-based CDSS where the user interactively characterizes her patient by browsing the system knowledge base to obtain the recommended treatment. We conducted a web-based evaluation of ASTI-GM as a before-after study to assess whether the system improves general practitioners' (GPs) performance and how they would use it. Five clinical cases had to be solved, as usual in the before phase, and using ASTI-GM in the after phase. On a 2-month period, 266 GPs participated and 1,981 prescription orders were collected. The overall guideline adherence rate increased from 27.2% to 64.3%. Only 56.4% of ASTI-GM uses corresponded to a "good use" of the system. Adherence increased from 28.5% to 86.1% in the sub-group of "good uses", whereas it only increased from 28.1% to 36.6% in the complementary sub-group. Reasons for non "good uses" of CDSSs should be investigated since they impede their potential impact.

Building clinical decision support systems requires a formalization of clinical practice guidelines (CPGs) including the verification of completeness to ensure all medically relevant situations are addressed. Recommendations that rely on completed knowledge cannot be but expert-based. Using French hypertension management guidelines, we characterized the status of a patient profile as evidence-based (EB), consensus-based (CB), or expert-based (XB). The distribution of these status on the formal patient profiles of ASTIGM knowledge base showed that 12.6% (0.5% EB and 12.1% CB) lead to explicit CPG recommendations. The same analysis on a sample of 435 actual patients medical records showed that 55% were covered by CPGs. The characterization of guideline-based CDSSs should be based on empirical data estimated from the target population of CPGs.

ASTI is a guideline-based decision support system to be used in primary care. We analyzed the knowledge modelling carried out in the development of ASTI knowledge base (KB) from French clinical practice guidelines (CPGs) on arterial hypertension management to evaluate the evidence status of therapeutic propositions issued by the system. We defined three status: "evidence-based" (EB) when propositions are graded A, B, or C, "consensus-based" (CB) when propositions are explicitly mentioned in CPGs but supported by professional agreement (grade D), and "non-supported" (NS) when propositions are expert-based and provided by a domain specialist. We compared the distributions of evidence status on the 44,571 theoretical patient profiles extracted from ASTI KB, and on a data set of 435 actual hypertensive patients. Only 8.3% of actual patients, managed by 0.5% of the KB, have an EB profile and 46.9% of patients, managed by 12.6% of the KB, have a CB profile. Thus, there is no CPG recommendation for nearly half of the patients (44.8% have a NS profile).

The ever growing pace at which medical knowledge is produced requires clinical practice guidelines (CPGs) to be regularly updated. Since clinical decision support systems (CDSSs) are effective means to implement guidelines in routine care, they have to be revised as their knowledge sources evolve. From one version to another, some parts are kept unchanged whereas others are more or less modified. We propose to characterize formally the different dimensions of recommendation evolution in two successive guideline versions from the knowledge modelling perspective. Each atomic recommendation is represented as a rule connecting a clinical condition to recommended action plans. Using subsumption-based comparisons, seven evolution patterns were identified: No change, Action plan refinement, New action plan, Condition refinement, Recommendation refinement, New practice, and Unmatched recommendation. The method has been evaluated on French bladder cancer guidelines in the revisions of 2002 and 2004.

In order to reduce practice variations and offer cancer patients the best treatments according to reference guidelines, therapeutic decisions have to be taken, in France, by "multidisciplinary staff meetings'' (MSMs) as patient-specific care plans which are then implemented by cancer specialists. OncoDoc2 is a CDSS implementing CancerEst guidelines, a "local reference guideline'', on breast cancer management. The system has been assessed in a pragmatic before/after study. The intervention consisted in the routine use of OncoDoc2 during MSMs of Tenon hospital. The MSM decision compliance rate with the reference guideline was significantly higher in the after period, increasing from 79% to 93%. MSM decision analysis showed that missing steps in treatment plans were the main cause of noncompliance during the before period. This cause was drastically reduced in the after period.

Establishing the diagnosis of drug-related pulmonary disease (DRPD) remains a difficult task because of the large number of drug-related toxic situations and the variety of clinical presentations. PneumoDoc is a computer-based support system designed to facilitate the diagnosis of lung disease using chronological, clinical, imaging, and cytological (alveolar lavage) input. These intrinsic items are crosschecked against extrinsic items reported in the literature (Pneumotox). Data input is in the form of yes-no questions. The final output displays the characteristic features of the observed clinical situation and calculates the probability of DRPD defined in five categories: incompatible, doubtful, compatible, suggestive, and highly suggestive. Use of multiple drugs, interaction with cardiopulmonary disease, and the absence of reported cases are limitations of the system.

Clinical Decision Support Systems (DSS) help improve health care quality. They usually incorporate an Execution Engine (EE), defined for each disease. We have designed, and present here, a generic execution engine, coupled with guideline-based disease specific rules stored in knowledge base (KB) as part of the prescription-critiquing mode of the ASTI project. This system was designed using two national guidelines for type 2 diabetes and hypertension. It takes into account the patient's clinical data, the tolerance and effectiveness of previous and current treatments and the physician's prescription made at the time. The functioning of the system has been speeded up and its maintenance made easier by indexing the KB rules according to the type of treatment they are linked to (e.g. monotherapy, etc.) and by classifying them into four categories. The EE's design formalizes generic therapeutic algorithms, leading to treatment options for cases of bad tolerance or insufficient effectiveness of the current treatment. Its applicability to other diseases was shown by applying it to dyslipidemia.

Computer-based decision support systems (CDSSs) are currently mostly reminder systems. However, the effectiveness of such systems to modify physician behavior is not always observed. We assume that this approach is appropriate when physicians think they know how to prescribe and consider they don't need to be helped, i.e. for simple clinical cases. On the opposite, on-demand approaches allowing for flexibility in the interpretation of patient conditions are more appropriate for more complex cases, e.g. in chronic disease management. ASTI is a CDSS operating in two modes, a critiquing mode working as a reminder-based system and a user-initiated guiding mode. Using a clinical case complexity score, a pre/post-intervention experiment with 10 GPs and 15 cases of hypertensive patients has been performed. Preliminary results tend to indicate that reminder-based interaction is appropriate for simple cases and that physicians are willing to use on-demand systems as clinical situations become complex, making both modes complementary.

Situations managed by clinical practice guidelines (CPGs) usually correspond to general descriptions of theoretical patients that suffer from only one disease in addition to the specific pathology CPGs focus on. The lack of decision support for complex multiple-disease patients is usually transferred to computer-based systems. Starting from the GEM-encoded instance of CPGs, we developed a module that automatically generated IF-THEN-WITH decision rules. A two-stage unification process has been implemented. All the rules whose IF-part is in partial matching with a patient clinical profile were triggered. A synthesis of triggered rules has then been performed to eliminate redundancies and incoherences. All remaining, eventually contradictory, recommendations were displayed to physicians leaving them the responsibility of handling the controversy and thus the opportunity to control the therapeutic decision.

The aim of this work is to develop an XML-based application for the automated generation of decision rules from a textual guideline encoded using the Guideline Elements Model (GEM). A formalization of guideline-based chronological steps of treatment has been proposed to resolve the semantic ambiguities of the original document. The GEM DTD has been extended in order to standardize both decision variable and action representations in recommendations. Under these assumptions, the 1999 Canadian Recommendations for the management of hypertension have been marked-up as a GEM-encoded instance of the extended DTD. An XML parser has been used to extract the relevant elements as IF and THEN clauses of decision rules. This GEM application generated 104 rules to be compared to the 98 rules manually developed from the same guideline during the ASTI project.