Search

Objective: to evaluate the usability of the “Systematic Review Support” computer system. Method: methodological study involving 21 students who evaluated the computer system in a remote and synchronous usability test. Two activities were carried out, containing 21 tasks simulating the initial steps of a review study, proposals for eliminating duplicate references, selecting studies and resolving disagreements between evaluators. At the end, participants answered the System Usability Scale questionnaire. The measures used were: completion rate with assistance (effectiveness) and time to complete the task (efficiency). The data was described in a table using the mean, standard deviation, minimum and maximum values. Results: in a total of 21 tasks, the participants had an assisted completion rate of 9% and an average time of 55.1 minutes (the expected time being 60 minutes). The average score provided by the System Usability Scale instrument was 82.4 (considering a score from 0 to 100), median 85; minimum value 35 and maximum 97.5. Conclusion: the computer system has the effectiveness and efficiency parameters required of a computer review tool. It has a usability score of 82.4 on a scale of 0 to 100 assessed by the System Usability Scale instrument. The version presented incorporates the adjustments identified in the test in use.

Resumo Objetivo Avaliar o desempenho do sistema web “Apoio à Revisão Sistemática” quanto à identificação de referências bibliográficas duplicadas, em comparação a outros programas. Métodos Trata-se de uma pesquisa metodológica que avalia o processo automático de identificação de duplicatas do sistema “Apoio à Revisão Sistemática” (versão 1.0), em comparação ao EndNote X9® e Rayyan® , considerando checagem manual como referência. Foi utilizado um conjunto de estudos relacionados a três temas sobre fibrose cística recuperados das bases de dados Pubmed, Embase e Web of Science. Para avaliação de desempenho, utilizaram-se a sensibilidade, especificidade, acurácia e área sob a curva ROC para cada software, em comparação à referência. Resultados As buscas nas bases de dados resultaram em 1332 estudos, sendo 273 (20,5%) verdadeiros duplicados. Em comparação aos dados de referência, o programa “Apoio à Revisão Sistemática” identificou maior proporção de duplicatas verdadeiras do que os demais. Os valores de sensibilidade, especificidade e acurácia do sistema “Apoio à Revisão Sistemática” apresentaram-se acima de 98%. Conclusão e implicações para a prática O sistema “Apoio à Revisão Sistemática” possui alta sensibilidade, especificidade e acurácia para identificação de estudos duplicados, otimizando o tempo e o trabalho dos revisores da área da saúde.

ABSTRACT Objective: To describe the main functions of the “Systematic Review Support” web-based system for removing duplicate articles and aiding eligibility analysis during the process of conducting systematic review studies. Methods: The system was developed based on the incremental build model using the Agile methodology. The software is proprietary source code and was published on a proprietary platform. The architecture of the production environment allows the infrastructure used to increase or decrease according to demand. The system functions are presented with insertion of screenshots of the interfaces of the version for personal computers during the simulation of a systematic review. Results: After importing the files containing the abstracts retrieved from the Pubmed, Embase, and Web of Science databases, the system identifies and removes duplicates for later reading and analysis of title and abstract, a stage which can be performed by one or more reviewers independently. After unblinding of reviewers, the decisions on the eligibility of the studies are compared automatically to help the researchers reach a consensus on any disagreements. Results can be filtered and a PDF produced containing the eligible studies. Conclusion: Version 1.0 of the system is available on the web (sysrev.azurewebsites.net) to assist researchers in the initial stages of systematic reviews.