Search

Background: The National Institute for Health and Clinical Excellence (NICE) single technology appraisal (STA) process was set up as a rapid way to appraise new technologies for use within the NHS in England and Wales and has been in place since 2005. Objectives: This study had five primary objectives: (1) to provide a map of the STA process to date; (2) to identify current approaches to the critical appraisal of manufacturers’ submissions (MSs) by Evidence Review Groups (ERGs); (3) to identify recurring themes in clarification letters sent to manufacturers; (4) to provide recommendations for possible alternative approaches to be used in the critical appraisal process; and (5) to revise the current ERG report template. Data sources: Data for the mapping of the STA process were obtained from the NICE website (www.nice.org.uk). Data for the analyses of the ERG reports and clarification letters were taken from the reports and letters themselves. Review methods: For the mapping, a spreadsheet was developed to collect data on 22 predefined variables related to timings and outcomes. Simple descriptive statistics were used to analyse the data. For the thematic analysis, a documentary analysis of 30 ERG reports was undertaken. Data on key elements of the MSs, the processes undertaken by ERGs and the strengths and weaknesses of MSs were extracted. A framework of a priori themes was developed. Data were extracted, coded and analysed according to a framework approach. Twenty-one clarification letters were examined and data were extracted using a set of codes to cover report quality, systematic review methods and clinical/economic issues. The current ERG report template was modified and sent to the current ERG teams for comment. All comments were considered and formed the basis for further revisions to the template. Results: Ninety-five STAs were included in the mapping exercise. Many STAs were subject to delay or cancellation for a variety of reasons. The ERG reports highlighted the strengths and weaknesses of MSs to the STA process. Thematic analysis of these data offered a means of clarifying and describing these aspects of the submissions. This analysis generated five themes: process, reporting, satisfaction of objectives, reliability and validity of findings, and content. Points from clarification letters were analysed and presented in four main categories: report quality, systematic review methods, clinical data analysis and economic data analysis. Limitations: Nearly all data were obtained from the NICE website; therefore, any errors in the data on the website will be reflected in the mapping analysis presented in this report. Missing data for the mapping exercise limit the generalisability of the findings. Analyses were limited to what was reported in the ERG reports and the clarification letters. Conclusions: Guidance suggested for manufacturers will help to ensure that more appropriate submissions are received in the future while recommendations provided for ERG teams will help to guide teams to ensure that reporting methods are transparent. Funding: The National Institute for Health Research Health Technology Assessment programme.

Background: Stickler syndrome, also known as hereditary progressive arthro-ophthalmopathy, is an inherited progressive disorder of the collagen connective tissues. Manifestations include short-sightedness, cataracts, retinal problems leading to retinal detachment and possible blindness. This is principally the case among individuals with type 1 Stickler Syndrome. It is the most commonly identified inherited cause of retinal detachment in childhood. However, there is no consensus regarding best practice and no current guidelines on prophylactic interventions for this population. Objectives: The aim of this systematic review was to assess the evidence for the clinical effectiveness and safety of primary prophylactic interventions for the prevention of retinal detachment in previously untreated eyes without retinal detachment in patients with Stickler syndrome. The primary outcome of interest was retinal detachment post prophylaxis. Data sources: A systematic search was made of 11 databases of published and unpublished literature, which included MEDLINE, MEDLINE In-Process & Other Non-Indexed Citations, EMBASE, the Cumulative Index to Nursing and Allied Health Literature and The Cochrane Library. There was no restriction by language or date. The references of all included studies were checked for further relevant citations and authors of studies with potentially relevant data were also contacted. Review methods: Two reviewers double-screened all titles and abstracts of the citations retrieved by the search to identify studies that satisfied the inclusion criteria. Both reviewers also independently extracted and quality assessed all included studies. A narrative synthesis was performed. Results: The literature search identified 1444 unique citations, of which four studies satisfied the inclusion criteria. The two principal studies were both retrospective cohort studies with control groups in populations with type 1 Stickler syndrome. One study evaluated 360° cryotherapy (n = 204) and the other focal or circumferential laser treatment (n = 22). Both studies reported a statistically significant difference in the rate of retinal detachment per eye between the groups receiving prophylaxis and the controls. However, both studies were subject to a high risk of bias. The results of the two supporting studies of Wagner–Stickler patients were either relatively inconsistent or unreliable. No study reported any major or long-term complications associated with the interventions. Despite the weaknesses of the evidence, the rate of retinal detachment in the intervention groups, especially the cryotherapy group, was lower than the rate either experienced in the study control groups or reported in other studies of untreated Stickler syndrome populations not exposed to prophylaxis. Conclusions: Only 360° cryotherapy and focal and circumferential laser treatment have been evaluated for the type 1 Stickler syndrome population, and then only by a single retrospective, controlled, cohort study in each case. Both of these studies report a significant difference between intervention and control groups (principally no treatment) and no major or long-term side effects or complications. However, there is a high risk of bias within these two studies, so the relative effectiveness of either intervention is uncertain. Future work: A service priority is to determine reliably the prevalence of Stickler syndrome, i.e. how many individuals have type 1 or type 2 Stickler syndrome, and their risk of retinal detachment and subsequent blindness. A non-randomised, prospective cohort comparison study, in which eligible participants are treated, followed-up and analysed in one of three study arms, for no treatment, laser therapy or cryotherapy, would potentially offer further certainty in terms of the relative efficacy of both prophylaxis versus no prophylaxis and cryotherapy versus laser therapy than is possible with the currently available data. Alternatively, continued follow-up and analysis of existing study data, and data collection from relevant sample populations, are required to assess the long-term risks of blindness, retinal detachment and prophylaxis. Funding: This study was funded by the National Institute for Health Research Health Technology Assessment programme.

Background: Health policy decisions must be relevant, evidence-based and transparent. Decision-analytic modelling supports this process but its role is reliant on its credibility. Errors in mathematical decision models or simulation exercises are unavoidable but little attention has been paid to processes in model development. Numerous error avoidance/identification strategies could be adopted but it is difficult to evaluate the merits of strategies for improving the credibility of models without first developing an understanding of error types and causes. Objectives: The study aims to describe the current comprehension of errors in the HTA modelling community and generate a taxonomy of model errors. Four primary objectives are to: (1) describe the current understanding of errors in HTA modelling; (2) understand current processes applied by the technology assessment community for avoiding errors in development, debugging and critically appraising models for errors; (3) use HTA modellers’ perceptions of model errors with the wider non-HTA literature to develop a taxonomy of model errors; and (4) explore potential methods and procedures to reduce the occurrence of errors in models. It also describes the model development process as perceived by practitioners working within the HTA community. Data sources: A methodological review was undertaken using an iterative search methodology. Exploratory searches informed the scope of interviews; later searches focused on issues arising from the interviews. Searches were undertaken in February 2008 and January 2009. In-depth qualitative interviews were performed with 12 HTA modellers from academic and commercial modelling sectors. Review methods: All qualitative data were analysed using the Framework approach. Descriptive and explanatory accounts were used to interrogate the data within and across themes and subthemes: organisation, roles and communication; the model development process; definition of error; types of model error; strategies for avoiding errors; strategies for identifying errors; and barriers and facilitators. Results: There was no common language in the discussion of modelling errors and there was inconsistency in the perceived boundaries of what constitutes an error. Asked about the definition of model error, there was a tendency for interviewees to exclude matters of judgement from being errors and focus on ‘slips’ and ‘lapses’, but discussion of slips and lapses comprised less than 20% of the discussion on types of errors. Interviewees devoted 70% of the discussion to softer elements of the process of defining the decision question and conceptual modelling, mostly the realms of judgement, skills, experience and training. The original focus concerned model errors, but it may be more useful to refer to modelling risks. Several interviewees discussed concepts of validation and verification, with notable consistency in interpretation: verification meaning the process of ensuring that the computer model correctly implemented the intended model, whereas validation means the process of ensuring that a model is fit for purpose. Methodological literature on verification and validation of models makes reference to the Hermeneutic philosophical position, highlighting that the concept of model validation should not be externalised from the decision-makers and the decision-making process. Interviewees demonstrated examples of all major error types identified in the literature: errors in the description of the decision problem, in model structure, in use of evidence, in implementation of the model, in operation of the model, and in presentation and understanding of results. The HTA error classifications were compared against existing classifications of model errors in the literature. A range of techniques and processes are currently used to avoid errors in HTA models: engaging with clinical experts, clients and decision-makers to ensure mutual understanding, producing written documentation of the proposed model, explicit conceptual modelling, stepping through skeleton models with experts, ensuring transparency in reporting, adopting standard housekeeping techniques, and ensuring that those parties involved in the model development process have sufficient and relevant training. Clarity and mutual understanding were identified as key issues. However, their current implementation is not framed within an overall strategy for structuring complex problems. Limitations: Some of the questioning may have biased interviewees responses but as all interviewees were represented in the analysis no rebalancing of the report was deemed necessary. A potential weakness of the literature review was its focus on spreadsheet and program development rather than specifically on model development. It should also be noted that the identified literature concerning programming errors was very narrow despite broad searches being undertaken. Conclusions: Published definitions of overall model validity comprising conceptual model validation, verification of the computer model, and operational validity of the use of the model in addressing the real-world problem are consistent with the views expressed by the HTA community and are therefore recommended as the basis for further discussions of model credibility. Such discussions should focus on risks, including errors of implementation, errors in matters of judgement and violations. Discussions of modelling risks should reflect the potentially complex network of cognitive breakdowns that lead to errors in models and existing research on the cognitive basis of human error should be included in an examination of modelling errors. There is a need to develop a better understanding of the skills requirements for the development, operation and use of HTA models. Interaction between modeller and client in developing mutual understanding of a model establishes that model’s significance and its warranty. This highlights that model credibility is the central concern of decision-makers using models so it is crucial that the concept of model validation should not be externalised from the decision-makers and the decision-making process. Recommendations for future research would be studies of verification and validation; the model development process; and identification of modifications to the modelling process with the aim of preventing the occurrence of errors and improving the identification of errors in models.