Back to Search
Start Over
Revolutionizing subjective assessments: A three-pronged comprehensive approach with NLP and deep learning.
- Source :
-
Expert Systems with Applications . Apr2024, Vol. 239, pN.PAG-N.PAG. 1p. - Publication Year :
- 2024
-
Abstract
- The enhanced answer evaluation system is a cutting-edge automated tool that evaluates subjective answers in various contexts, such as educational assessments, surveys, and feedback forms. The proposed system leverages Natural Language Processing (NLP) and deep learning techniques to analyse subjective answers and provide evaluation scores with precision. Students' answers are evaluated based on various criteria, such as keywords, context, relevance, coherence, and similarity. This paper introduces an architecture for a subjective answer evaluator using three main aspects: detection of keywords, similarity matrix, and presence of named entities. It combines the three aspects and provides a final score. It provides a standardized mechanism to score a given user answer compared to the particular model answer without human prejudice. This research aims to transcend traditional methodologies that predominantly utilize keyword or keyphrase scoring (text-based similarity) to determine the final score of an answer without delving into its technical intricacies. The semantic similarity (vector-based) employs vector data representations for score calculation. This approach necessitates partitioning data into multiple vectors for a comprehensive analysis. While text similarity is effective for short answers, its efficacy diminishes as the length of the answer increases. Therefore, this study emphasizes the critical role of similarity scoring and Named Entity Recognition (NER) scoring in evaluating more extended responses based on the stsb-en-main dataset (short answers) and a custom dataset with 190 records. This research reveals its remarkable performance, which excels through a dynamic three-pronged approach: keyword scoring, semantic similarity, and NER scoring with models like Yet Another Keyword Extractor (YAKE), SimCSE and Camembert. These three independent components synergize to produce unmatched results, establishing a new standard in the field. This enhancement led to Root Mean Square Error (RMSE) scores of 0.031 (optimized error rate) and an impressive 71%+ accuracy for our comprehensive system. This achievement surpasses existing works, which typically reached accuracies ranging between 40%–60% for long answers. • Evaluation using a three-pronged approach of Keyword, Semantic & NER scoring. • Semantic scores enhance similarity matrix grading and impact evaluation. • NER scores address proper noun issues. • Deep Neural Network predicts actual marks effectively. • The best-performing LLM models are YAKE, SimCSE and Camembert. [ABSTRACT FROM AUTHOR]
Details
- Language :
- English
- ISSN :
- 09574174
- Volume :
- 239
- Database :
- Academic Search Index
- Journal :
- Expert Systems with Applications
- Publication Type :
- Academic Journal
- Accession number :
- 174875364
- Full Text :
- https://doi.org/10.1016/j.eswa.2023.122470