Search

The digitization of operation and maintenance in the intelligent power grid equipment relies on a diverse array of information for smart decision-making. In the domain of intelligent decision generation, proficiency is contingent upon extensive learning from copious amounts of text. This necessitates not only robust processing capabilities but also a high level of specialization. In addressing situations where authorization is lacking, pre-trained language models (PLMs) have already provided ideas when confronted with specialized domains or tasks. In consideration of the complexity of textual content in the field of the power grid, which encompasses a multitude of specialized knowledge and involves an abundance of proprietary terminology, we have undertaken an exploration of pre-trained model specialization using the power grid domain as an example, specifically for the task of generating maintenance strategies. A two-stage fine-tuning approach (P2FT) is employed, utilizing a large-scale pre-training model specifically designed for natural language processing. The efficacy and practical value of this method were evaluated through multiple metrics, juxtaposed with other advanced approaches involving low-parameter or parameter-free fine-tuning methods. Through a meticulous analysis and validation of experimental outcomes, we have corroborated the feasibility and practical application value of employing this approach for pre-trained model specialization. Additionally, it has furnished valuable guidance for text generation within both the Chinese language domain and the power grid domain. [ABSTRACT FROM AUTHOR]

As consumer electronics evolve towards greater intelligence, their automation and complexity also increase, making it difficult for users to diagnose faults when they occur. To address the problem where users, relying solely on their own knowledge, struggle to diagnose faults in consumer electronics promptly and accurately, we propose a multimodal knowledge graph-based text generation method. Our method begins by using deep learning models like the Residual Network (ResNet) and Bidirectional Encoder Representations from Transformers (BERT) to extract features from user-provided fault information, which can include images, text, audio, and even olfactory data. These multimodal features are then combined to form a comprehensive representation. The fused features are fed into a graph convolutional network (GCN) for fault inference, identifying potential fault nodes in the electronics. These fault nodes are subsequently fed into a pre-constructed knowledge graph to determine the final diagnosis. Finally, this information is processed through the Bias-term Fine-tuning (BitFit) enhanced Chinese Pre-trained Transformer (CPT) model, which generates the final fault diagnosis text for the user. The experimental results show that our proposed method achieves a 4.4% improvement over baseline methods, reaching a fault diagnosis accuracy of 98.4%. Our approach effectively leverages multimodal fault information, addressing the challenges users face in diagnosing faults through the integration of graph convolutional network and knowledge graph technologies. [ABSTRACT FROM AUTHOR]

This study addresses the challenge of automating the creation of Exchange Information Requirements (EIRs) for construction projects using Building Information Modelling (BIM) and Digital Twins, as specified in the ISO 19650 standard. This paper focuses on automating the classification of EIR paragraphs according to the ISO 19650 standard's categories, aiming to improve information management in construction projects. It addresses a gap in applying AI to enhance BIM project management, where barriers often include technological limitations, a shortage of specialists, and limited understanding of the methodology. The proposed method uses Word2Vec for text vectorisation and Support Vector Machines (SVMs) with an RBF kernel for text classification, and it attempts to apply Word2Vec with cosine similarity for text generation. The model achieved an average F1 score of 0.7, with predicted categories for provided sentences and similar matches for selected phrases. While the text classification results were promising, further refinement is required for the text generation component. This study concludes that integrating AI tools such as Word2Vec and SVM offers a feasible solution for enhancing EIR creation. However, further development of text generation, particularly using advanced techniques such as GPT, is recommended. These findings contribute to improving managing complex construction projects and advancing digitalization in the AECO sector. [ABSTRACT FROM AUTHOR]

This research presents an approach to automatic multiple-choice question (MCQ) generation in the Spanish language, using mT5-based models. The process encompasses three crucial tasks: candidate answer extraction, answer-aware question generation, and distractor generation. A methodical pipeline is structured to seamlessly integrate these tasks, converting an input text into a systematic questionnaire. For model fine-tuning, the Stanford Question Answering Dataset is employed for the first two tasks, while a combination of three different multiple-choice question datasets, translated automatically into Spanish, is used for the distractor generation task. The efficiency of the models is then evaluated by using a triad of metrics, namely BLEU, ROUGE-L, and cosine similarity. The outcomes indicate a marginal deviation from the baseline model in the question generation task but demonstrate superior performance in the distractor generation task. Importantly, this research emphasizes the potential and effectiveness of language models for automating MCQ generation, providing a valuable contribution to the field and enhancing the understanding and application of such models in the context of the Spanish language. [ABSTRACT FROM AUTHOR]

Efficiently leveraging semantic information is crucial for advancing video captioning in recent years. But, prevailing approaches that involve designing various Part-of-Speech (POS) tags as prior information lack essential linguistic knowledge guidance throughout the training procedure, particularly in the context of POS and initial description generation. Furthermore, the restriction to a single source of semantic information ignores the potential for varied interpretations inherent in each video. To solve these problems, we propose the Exploring Deeper into Semantics (EDS) method for video captioning. EDS comprises three feasible modules that focus on semantic information. Specifically, we propose the Semantic Supervised Generation (SSG) module. It integrates semantic information as a prior, and facilitates enriched interrelations among words for POS supervision. A novel Similarity Semantic Extension (SSE) module is proposed to employ a query-based semantic expansion for collaboratively generating fine-grained content. Additionally, the proposed Input Semantic Enhancement (ISE) module provides a strategy for mitigating the information constraints faced during the initial phase of word generation. The experiments conducted show that, by exploiting semantic information through supervision, extension, and enhancement, EDS not only yields promising results but also underlines the effectiveness. Code will be available at https://github.com/BradenJoson/EDS. • A novel method EDS is proposed to explore semantics utilization for video captioning. • A prior-based extractor that provides accurate semantic information. • A similarity-based extension module that handles varied video interpretations. • A simple module that refines the initial word generation with enhanced semantics. [ABSTRACT FROM AUTHOR]

Missing data in tabular dataset is a common issue as the performance of downstream tasks usually depends on the completeness of the training dataset. Previous missing data imputation methods focus on numeric and categorical columns, but we propose a novel end-to-end approach called Table Transformers for Imputing Textual Attributes (TTITA) based on the transformer to impute unstructured textual columns using other columns in the table. We conduct extensive experiments on three datasets, and our approach shows competitive performance outperforming baseline models such as recurrent neural networks and Llama2. The performance improvement is more significant when the target sequence has a longer length. Additionally, we incorporate multi-task learning to simultaneously impute for heterogeneous columns, boosting the performance for text imputation. We also qualitatively compare with ChatGPT for realistic applications. • Proposed TTITA to impute text attributes given other heterogeneous tabular columns. • Encoded inputs into a context vector for cross-attention in the transformer decoder. • Outperformed baseline models including the GRU and Llama2 on real-world datasets. • Incorporated multi-task learning for multi-column imputation and boosting performance. • Prepared the software as an open-source package for custom applications. [ABSTRACT FROM AUTHOR]

Abstract The digitization of operation and maintenance in the intelligent power grid equipment relies on a diverse array of information for smart decision-making. In the domain of intelligent decision generation, proficiency is contingent upon extensive learning from copious amounts of text. This necessitates not only robust processing capabilities but also a high level of specialization. In addressing situations where authorization is lacking, pre-trained language models (PLMs) have already provided ideas when confronted with specialized domains or tasks. In consideration of the complexity of textual content in the field of the power grid, which encompasses a multitude of specialized knowledge and involves an abundance of proprietary terminology, we have undertaken an exploration of pre-trained model specialization using the power grid domain as an example, specifically for the task of generating maintenance strategies. A two-stage fine-tuning approach (P2FT) is employed, utilizing a large-scale pre-training model specifically designed for natural language processing. The efficacy and practical value of this method were evaluated through multiple metrics, juxtaposed with other advanced approaches involving low-parameter or parameter-free fine-tuning methods. Through a meticulous analysis and validation of experimental outcomes, we have corroborated the feasibility and practical application value of employing this approach for pre-trained model specialization. Additionally, it has furnished valuable guidance for text generation within both the Chinese language domain and the power grid domain.

Multi-turn dialogue generation tasks heavily rely on capturing contextual information. However, in real-life scenarios, capturing the speaker’s needs accurately cannot be achieved solely with limited context, so background knowledge information is also necessary. Existing works focus on using local keywords to retrieve external knowledge and simply concatenating retrieval information with context, which results in low-quality retrieved external knowledge and redundant context, leading to difficulty in understanding the context. To address these issues, this paper proposes the CRKG model. The CRKG mode first designs a turn-level attention mechanism to capture important information in the context. Then, it retrieves knowledge from historical dialogues as an external knowledge-base based on the important information representation. Finally, it designs a hierarchical fusion encoder to dynamically integrate the retrieved information. We validate our proposed method on text-based small parameter size model and large language model. Experimental results show that our proposed method achieves the best results on multiple public datasets. [ABSTRACT FROM AUTHOR]

Transformer-based language models, including ChatGPT, have demonstrated exceptional performance in various natural language generation tasks. However, there has been limited research evaluating ChatGPT’s keyphrase generation ability, which involves identifying informative phrases that accurately reflect a document’s content. This study seeks to address this gap by comparing ChatGPT’s keyphrase generation performance with state-of-the-art models, while also testing its potential as a solution for two significant challenges in the field: domain adaptation and keyphrase generation from long documents. We conducted experiments on eight publicly available datasets spanning scientific, news, and biomedical domains, analyzing performance across both short and long documents. Our results show that ChatGPT outperforms current state-of-the-art models in all tested datasets and environments, generating high-quality keyphrases that adapt well to diverse domains and document lengths. [ABSTRACT FROM AUTHOR]

A text generation based multidisciplinary problem identification method is proposed, which does not rely on a large amount of data annotation.

Sentiment analysis is a vital tool in natural language processing (NLP), enabling the interpretation and understanding of opinions expressed in textual data. Traditional sentiment analysis methods, often limited to document or sentence-level analysis, primarily focus on identifying the sentiment without generating detailed sentiment text expressions. To address this limitation, we propose a novel Aspect-Specific Sentiment Expression Generation (ASSEG) model. Unlike traditional approaches, the ASSEG model leverages advanced text generation models, such as GPT-2 and T5, to automatically generate sentiment expressions tailored to diverse aspects of entities discussed in the text. The key innovation of our approach lies in the integration of aspect-specific attention mechanisms, which enable the model to effectively identify and prioritize aspects within the text, generating coherent and contextually relevant sentiment expressions. Our methodology includes using Recurrent Generative Adversarial Networks (RGANs) for data augmentation, addressing data imbalance issues, and enhancing the robustness of sentiment analysis models. Experimental evaluations were conducted on domain-specific datasets, including laptop and restaurant reviews. Our experimental evaluations on domain-specific datasets, including laptop and restaurant reviews, demonstrate the superior performance of our ASSEG model. The GPT-2 model achieved an accuracy of 75% and 65%, and an F1 score of 77% and 65% for restaurant and laptop datasets, respectively. Meanwhile, the T5 model outperformed GPT-2, achieving an accuracy of 85% and 75%, and an F1 score of 83% and 74% for restaurant and laptop datasets, respectively. These results highlight the potential of the ASSEG model, offering deeper insights into user opinions by generating detailed and contextually relevant sentiment expressions. [ABSTRACT FROM AUTHOR]

Objectives To enhance the performance of large language models (LLMs) in biomedical natural language processing (BioNLP) by introducing a domain-specific instruction dataset and examining its impact when combined with multi-task learning principles. Materials and Methods We created the BioInstruct , comprising 25 005 instructions to instruction-tune LLMs (LLaMA 1 and 2, 7B and 13B version). The instructions were created by prompting the GPT-4 language model with 3-seed samples randomly drawn from an 80 human curated instructions. We employed Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. We then evaluated these instruction-tuned LLMs on several BioNLP tasks, which can be grouped into 3 major categories: question answering (QA), information extraction (IE), and text generation (GEN). We also examined whether categories (eg, QA, IE, and generation) of instructions impact model performance. Results and Discussion Comparing with LLMs without instruction-tuned, our instruction-tuned LLMs demonstrated marked performance gains: 17.3% in QA on average accuracy metric, 5.7% in IE on average F1 metric, and 96% in Generation tasks on average GPT-4 score metric. Our 7B-parameter instruction-tuned LLaMA 1 model was competitive or even surpassed other LLMs in the biomedical domain that were also fine-tuned from LLaMA 1 with vast domain-specific data or a variety of tasks. Our results also show that the performance gain is significantly higher when instruction fine-tuning is conducted with closely related tasks. Our findings align with the observations of multi-task learning, suggesting the synergies between 2 tasks. Conclusion The BioInstruct dataset serves as a valuable resource and instruction tuned LLMs lead to the best performing BioNLP applications. [ABSTRACT FROM AUTHOR]

Objective To solve major clinical natural language processing (NLP) tasks using a unified text-to-text learning architecture based on a generative large language model (LLM) via prompt tuning. Methods We formulated 7 key clinical NLP tasks as text-to-text learning and solved them using one unified generative clinical LLM, GatorTronGPT, developed using GPT-3 architecture and trained with up to 20 billion parameters. We adopted soft prompts (ie, trainable vectors) with frozen LLM, where the LLM parameters were not updated (ie, frozen) and only the vectors of soft prompts were updated, known as prompt tuning. We added additional soft prompts as a prefix to the input layer, which were optimized during the prompt tuning. We evaluated the proposed method using 7 clinical NLP tasks and compared them with previous task-specific solutions based on Transformer models. Results and Conclusion The proposed approach achieved state-of-the-art performance for 5 out of 7 major clinical NLP tasks using one unified generative LLM. Our approach outperformed previous task-specific transformer models by ∼3% for concept extraction and 7% for relation extraction applied to social determinants of health, 3.4% for clinical concept normalization, 3.4%-10% for clinical abbreviation disambiguation, and 5.5%-9% for natural language inference. Our approach also outperformed a previously developed prompt-based machine reading comprehension (MRC) model, GatorTron-MRC, for clinical concept and relation extraction. The proposed approach can deliver the "one model for all" promise from training to deployment using a unified generative LLM. [ABSTRACT FROM AUTHOR]

Copyright of Procesamiento del Lenguaje Natural is the property of Sociedad Espanola para el Procesamiento del Lenguaje Natural and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The text generation technique employs the transformation of the word document from the source to the targeted document based on the sequence to sequence generation. Video captioning, language identification, image captioning, recognition of speech, machine translation, and several other natural language generations are the application areas of the text generation techniques. The Electroencephalographic (EEG) signals record brain activity and are considered the source of information for using the brain-computer interface. Several kinds of research were developed for text generation. The most challenging task is more accurate text generation by considering the large contextual information and the significant features for generating the text. Hence, in this research, text generation using Folded deep learning is proposed for generating the text through text prediction and suggestion through the non-invasive technique. The EEG signal recorded from the patients is utilized for the prediction of the first letter using the proposed Folded Ensemble Deep convolutional neural network (DeepCNN), in which the hybrid ensemble activation function along with the folded concept in validating the training data to obtain the network stability and to solve the class imbalance issue. Then, the next letter suggestion is employed using the proposed Folded Ensemble Bidirectional long short-term memory (BiLSTM) approach based on the eye-blink criteria for generating the sequence-to-sequence text generation. The enhanced performance is evaluated using accuracy, precision, and recall and acquired the maximal values of 97.22%, 98.00%, and 98.00%, respectively. The proposed method can be utilized for real-time processing applications due to its non-invasive nature. [ABSTRACT FROM AUTHOR]

Recently, Large Language Models (LLMs) have seen some impressive leaps, achieving the ability to accomplish several tasks, from text completion to powerful chatbots. The great variety of available LLMs and the fast pace of technological innovations in this field, is making LLM assessment a hard task to accomplish: understanding not only what such a kind of systems generate but also which is the quality of their results is of a paramount importance. Generally, the quality of a synthetically generated object could refer to the reliability of the content, to the lexical variety or coherence of the text. Regarding the quality of text generation, an aspect that up to now has not been adequately discussed is concerning the readability of textual artefacts. This work focuses on the latter aspect, proposing a set of experiments aiming to better understanding and evaluating the degree of readability of texts automatically generated by an LLM. The analysis is performed through an empirical study based on: considering a subset of five pre-trained LLMs; considering a pool of English text generation tasks, with increasing difficulty, assigned to each of the models; and, computing a set of the most popular readability indexes available from the computational linguistics literature. Readability indexes will be computed for each model to provide a first perspective of the readability of textual contents artificially generated can vary among different models and under different requirements of the users. The results obtained by evaluating and comparing different models provide interesting insights, especially into the responsible use of these tools by both beginners and not overly experienced practitioners. [ABSTRACT FROM AUTHOR]

In the fast-paced realm of digital transformation, businesses are increasingly pressured to innovate and boost efficiency to remain competitive and foster growth. Large Language Models (LLMs) have emerged as game-changers across industries, revolutionizing various sectors by harnessing extensive text data to analyze and generate human-like text. Despite their impressive capabilities, LLMs often encounter challenges when dealing with domain-specific queries, potentially leading to inaccuracies in their outputs. In response, Retrieval-Augmented Generation (RAG) has emerged as a viable solution. By seamlessly integrating external data retrieval into text generation processes, RAG aims to enhance the accuracy and relevance of the generated content. However, existing literature reviews tend to focus primarily on the technological advancements of RAG, overlooking a comprehensive exploration of its applications. This paper seeks to address this gap by providing a thorough review of RAG applications, encompassing both task-specific and discipline-specific studies, while also outlining potential avenues for future research. By shedding light on current RAG research and outlining future directions, this review aims to catalyze further exploration and development in this dynamic field, thereby contributing to ongoing digital transformation efforts. [ABSTRACT FROM AUTHOR]

Copyright of Inonu University Journal of the Faculty of Education (INUJFE) is the property of Inonu University Journal of the Faculty of Education and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

There is a pressing need for healthcare conversational agents with domain-specific expertise to ensure the provision of accurate and reliable information tailored to specific medical contexts. Moreover, there is a notable gap in research ensuring the credibility and trustworthiness of the information provided by these healthcare agents, particularly in critical scenarios such as medical emergencies. Pilgrims come from diverse cultural and linguistic backgrounds, often facing difficulties in accessing medical advice and information. Establishing an AI-powered multilingual chatbot can bridge this gap by providing readily available medical guidance and support, contributing to the well-being and safety of pilgrims. In this paper, we present a comprehensive methodology aimed at enhancing the reliability and efficacy of healthcare conversational agents, with a specific focus on addressing the needs of Hajj pilgrims. Our approach leverages domain-specific fine-tuning techniques on a large language model, alongside synthetic data augmentation strategies, to optimize performance in delivering contextually relevant healthcare information by introducing the HajjHealthQA dataset. Additionally, we employ a retrieval-augmented generation (RAG) module as a crucial component to validate uncertain generated responses, which improves model performance by 5%. Moreover, we train a secondary AI agent on a well-known health fact-checking dataset and use it to validate medical information in the generated responses. Our approach significantly elevates the chatbot's accuracy, demonstrating its adaptability to a wide range of pilgrim queries. We evaluate the chatbot's performance using quantitative and qualitative metrics, highlighting its proficiency in generating accurate responses and achieve competitive results compared to state-of-the-art models, in addition to mitigating the risk of misinformation and providing users with trustworthy health information. [ABSTRACT FROM AUTHOR]

In the realm of the video captioning field, acquiring large amounts of high-quality aligned video–text pairs remains laborious, impeding its practical applications. Therefore, we explore the modelling techniques for unsupervised video captioning. Using text inputs similar to the video representation to generate captions has been a successful unsupervised video captioning generation strategy in the past. However, this setting relies solely on the textual data for training, neglecting vital visual cues related to the spatio-temporal appearance within the video. The absence of visual information increases the risk of generating erroneous video captions. In view of this, we propose a novel unsupervised video captioning method that introduces visual information related to text features keywords to implicitly enhance training for text generation tasks. Simultaneously, our method incorporates sentence to explicitly augment the training process. our method injects additional implicit visual features and explicit keywords into the model, Which can inject the generated captions with more accurate semantics. the experimental analysis demonstrates the merit of the proposed formulation, achieving superior performance against the state-of-the-art unsupervised studies. • Contrast learning is used to minimise the disparities between pseudo-text labels and video features. • Visual clues are aligned with the text generator for consistent semantic enhancement. • Leveraging the Found within the sentences for semantic preservation. • Outperforming existing unsupervised video captioning approaches. [ABSTRACT FROM AUTHOR]

Generating diverse and factual text is challenging and is receiving increasing attention. By sampling from the latent space, variational autoencoder-based models have recently enhanced the diversity of generated text. However, existing research predominantly depends on summarization models to offer paragraph-level semantic information for enhancing factual correctness. The challenge lies in effectively generating factual text using sentence-level variational autoencoder-based models. In this paper, a novel model called fact-aware conditional variational autoencoder is proposed to balance the factual correctness and diversity of generated text. Specifically, our model encodes the input sentences and uses them as facts to build a conditional variational autoencoder network. By training a conditional variational autoencoder network, the model is enabled to generate text based on input facts. Building upon this foundation, the input text is passed to the discriminator along with the generated text. By employing adversarial training, the model is encouraged to generate text that is indistinguishable to the discriminator, thereby enhancing the quality of the generated text. To further improve the factual correctness, inspired by the natural language inference system, the entailment recognition task is introduced to be trained together with the discriminator via multi-task learning. Moreover, based on the entailment recognition results, a penalty term is further proposed to reconstruct the loss of our model, forcing the generator to generate text consistent with the facts. Experimental results demonstrate that compared with competitive models, our model has achieved substantial improvements in both the quality and factual correctness of the text, despite only sacrificing a small amount of diversity. Furthermore, when considering a comprehensive evaluation of diversity and quality metrics, our model has also demonstrated the best performance. [ABSTRACT FROM AUTHOR]

Deep learning is revolutionizing radiology report generation (RRG) with the adoption of vision encoder–decoder (VED) frameworks, which transform radiographs into detailed medical reports. Traditional methods, however, often generate reports of limited diversity and struggle with generalization. Our research introduces reinforcement learning and text augmentation to tackle these issues, significantly improving report quality and variability. By employing RadGraph as a reward metric and innovating in text augmentation, we surpass existing benchmarks like BLEU4, ROUGE-L, F1CheXbert, and RadGraph, setting new standards for report accuracy and diversity on MIMIC-CXR and Open-i datasets. Our VED model achieves F1-scores of 66.2 for CheXbert and 37.8 for RadGraph on the MIMIC-CXR dataset, and 54.7 and 45.6 , respectively, on Open-i. These outcomes represent a significant breakthrough in the RRG field. The findings and implementation of the proposed approach, aimed at enhancing diagnostic precision and radiological interpretations in clinical settings, are publicly available on GitHub to encourage further advancements in the field. [ABSTRACT FROM AUTHOR]

Text style transfer aims to rephrase a sentence to match the desired style while retaining the original content. As a controllable text generation task, mainstream approaches use content-independent style embedding as control variables to guide stylistic generation. Nonetheless, stylistic properties are context-sensitive even under the same style. For example, “delicious” and “helpful” convey positive sentiments, although they are more likely to describe food and people, respectively. Therefore, desired style signals must vary with the content. To this end, we propose a memory-enhanced transfer method, which learns fine-grained style representation concerning content to assist transfer. Rather than employing static style embedding or latent variables, our method abstracts linguistic characteristics from training corpora and memorizes subdivided content with the corresponding style representations. The style signal is dynamically retrieved from memory using the content as a query, providing a more expressive and flexible latent style space. To address the imbalance between quantity and quality in different content, we further introduce a calibration method to augment memory construction by modeling the relationship between candidate styles. Experimental results obtained using three benchmark datasets confirm the superior performance of our model compared to competitive approaches. The evaluation metrics and case study also indicate that our model can generate diverse stylistic phrases matching context. [ABSTRACT FROM AUTHOR]

In this study, we primarily aim to address the exposure bias issue in long text generation intrinsic to statistical language models. We propose a sentence-level heuristic tree search algorithm, specially tailored for long text generation, to mitigate the problem by managing generated texts in a tree structure and curbing the compounding of biases. Our algorithm utilizes two pre-trained language models, an auto-regressive model for generating new sentences and an auto-encoder model for evaluating sentence quality. These models work in tandem to perform four critical operations: expanding the text tree with new sentences, evaluating the quality of the additions, sampling potential unfinished text fragments for further generation, and pruning leaf nodes deemed unpromising. This iterative process continues until a pre-defined number of [EOS] tokens are produced, at which point we select the highest-scoring completed text as our final output. Moreover, we pioneer two novel token-level decoding techniques—nucleus sampling with temperature and diverse beam search with sampling. These methods, integrated with our sentence-level search algorithm, aim to improve the consistency and diversity of text generation. Experimental results, both automated measures (including Jaccard similarity, Word2vec similarity, and unique word ratio) and human evaluations (assessing consistency, fluency, and rhetorical skills), conclusively demonstrate that our approach considerably enhances the quality of machine-generated long-form text. Through this research, we aim to inspire further innovations in sentence-level search-based text generation algorithms. [ABSTRACT FROM AUTHOR]

As consumer electronics evolve towards greater intelligence, their automation and complexity also increase, making it difficult for users to diagnose faults when they occur. To address the problem where users, relying solely on their own knowledge, struggle to diagnose faults in consumer electronics promptly and accurately, we propose a multimodal knowledge graph-based text generation method. Our method begins by using deep learning models like the Residual Network (ResNet) and Bidirectional Encoder Representations from Transformers (BERT) to extract features from user-provided fault information, which can include images, text, audio, and even olfactory data. These multimodal features are then combined to form a comprehensive representation. The fused features are fed into a graph convolutional network (GCN) for fault inference, identifying potential fault nodes in the electronics. These fault nodes are subsequently fed into a pre-constructed knowledge graph to determine the final diagnosis. Finally, this information is processed through the Bias-term Fine-tuning (BitFit) enhanced Chinese Pre-trained Transformer (CPT) model, which generates the final fault diagnosis text for the user. The experimental results show that our proposed method achieves a 4.4% improvement over baseline methods, reaching a fault diagnosis accuracy of 98.4%. Our approach effectively leverages multimodal fault information, addressing the challenges users face in diagnosing faults through the integration of graph convolutional network and knowledge graph technologies.

Diffusion models have achieved tremendous success in modeling continuous data modalities, such as images, audio, and video, yet their application in discrete data domains (e.g., natural language) has been limited. Existing methods primarily represent discrete text in a continuous diffusion space, incurring significant computational overhead during training and resulting in slow sampling speeds. This paper introduces LaDiffuSeq, a latent diffusion-based text generation model incorporating an encoder–decoder structure. Specifically, it first employs a pretrained encoder to map sequences composed of attributes and corresponding text into a low-dimensional latent vector space. Then, without the guidance of a classifier, it performs the diffusion process for the sequence's corresponding latent space. Finally, a pretrained decoder is used to decode the newly generated latent vectors, producing target texts that are relevant to themes and possess multiple emotional granularities. Compared to the benchmark model, DiffuSeq, this model achieves BERTScore improvements of 0.105 and 0.009 on two public real-world datasets (ChnSentiCorp and a debate dataset), respectively; perplexity falls by 3.333 and 4.562; and it effectively quadruples the text generation sampling speed. [ABSTRACT FROM AUTHOR]

Chinese medicine (CM) diagnosis intellectualization is one of the hotspots in the research of CM modernization. The traditional CM intelligent diagnosis models transform the CM diagnosis issues into classification issues, however, it is difficult to solve the problems such as excessive or similar categories. With the development of natural language processing techniques, text generation technique has become increasingly mature. In this study, we aimed to establish the CM diagnosis generation model by transforming the CM diagnosis issues into text generation issues. The semantic context characteristic learning capacity was enhanced referring to Bidirectional Long Short-Term Memory (BILSTM) with Transformer as the backbone network. Meanwhile, the CM diagnosis generation model Knowledge Graph Enhanced Transformer (KGET) was established by introducing the knowledge in medical field to enhance the inferential capability. The KGET model was established based on 566 CM case texts, and was compared with the classic text generation models including Long Short-Term Memory sequence-to-sequence (LSTM-seq2seq), Bidirectional and Auto-Regression Transformer (BART), and Chinese Pre-trained Unbalanced Transformer (CPT), so as to analyze the model manifestations. Finally, the ablation experiments were performed to explore the influence of the optimized part on the KGET model. The results of Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation 1 (ROUGE1), ROUGE2 and Edit distance of KGET model were 45.85, 73.93, 54.59 and 7.12, respectively in this study. Compared with LSTM-seq2seq, BART and CPT models, the KGET model was higher in BLEU, ROUGE1 and ROUGE2 by 6.00–17.09, 1.65–9.39 and 0.51–17.62, respectively, and lower in Edit distance by 0.47–3.21. The ablation experiment results revealed that introduction of BILSTM model and prior knowledge could significantly increase the model performance. Additionally, the manual assessment indicated that the CM diagnosis results of the KGET model used in this study were highly consistent with the practical diagnosis results. In conclusion, text generation technology can be effectively applied to CM diagnostic modeling. It can effectively avoid the problem of poor diagnostic performance caused by excessive and similar categories in traditional CM diagnostic classification models. CM diagnostic text generation technology has broad application prospects in the future. [ABSTRACT FROM AUTHOR]

This study sought to examine the influence of transcription skills, evaluated using graphonomic measures, on the proficiency of text generation in students attending primary schools in Spain. A longitudinal design was employed involving 278 Spanish students distributed across three cohorts (cohort 1: 1st-2nd-4th grade; cohort 2: 2nd-3rd-5th grade; and cohort 3: 3rd-4th-6th grade). Two data collection points were used to administer the graphonomic measures, and a composition letter task was conducted at the conclusion of the study. Four multigroup structural equation models were employed, examining the direct pathways from graphonomic measures (i.e., pressure, speed, pauses, and road length) on text generation (i.e., length, fluency, planning, revision, and organization). The models demonstrated a good fit to the data. The findings from the four models, analyzed within the three cohorts, indicated that the significant effect of transcription (i.e., handwriting) on text production was primarily observed in Cohort 1 (early grades), while no significant effects were found in Cohort 2 (intermediate grades). This suggests that the importance of handwriting in text production in a transparent orthography may be more pronounced during the initial stages of writing development when students are acquiring foundational writing skills. [ABSTRACT FROM AUTHOR]

This paper deals with automated football match reports as a common genre of automated journalism. Based on a corpus of automated and human-written reports (n = 1,302) on the same set of matches and with reference to linguistic concepts of text and textuality, the textual properties of these texts are analyzed both quantitatively and qualitatively. The analysis is based on the idea that the task of text generation can be described as the task of automatically selecting cues of textuality such as connectives or signals of thematic relatedness. The results show that automated and human-written texts differ significantly in the use of these cues, particularly in the use of linguistic means for creating evaluation and contrast, and thus allow to trace in detail, how these cues contribute to cohesion, coherence and narrative qualities. Different from computational linguistic approaches focused on optimizing text generation algorithms, this paper proposes to use automated texts, which are to some extent imperfect, as models of textuality that through their imperfection can say something about the nature of texts in general. The paper thus contributes to the field of (mostly communication studies) research on automated journalism in which the texts themselves are rarely investigated. [ABSTRACT FROM AUTHOR]

This study addresses the challenge of class imbalance in sentiment analysis by utilizing synthetic data to balance training datasets. We introduce an innovative approach using the GPT-3 model's sentence-by-sentence generation technique to generate synthetic data, specifically targeting underrepresented negative and neutral sentiments. Our method aims to align these minority classes with the predominantly positive sentiment class in a Coursera course review dataset, with the goal of enhancing the performance of sentiment classification. This research demonstrates that our proposed method successfully enhances sentiment classification performance, as evidenced by improved accuracy and F1-score metrics across five deep-learning models. However, when compared to our previous research utilizing fine-tuning techniques, the current method shows a relative shortfall. The fine-tuning approach yields better results in all models tested, indicating the importance of data novelty and diversity in synthetic data generation. In terms of the deep-learning model used for classification, the notable finding is the significant performance improvement of the Recurrent Neural Network (RNN) model compared to other models like CNN, LSTM, BiLSTM, and GRU, highlighting the impact of the model choice and architecture depth. This study emphasizes the critical role of synthetic data quality and strategic deep-learning model implementation in sentiment analysis. The results suggest that the careful consideration of training data and model attributes is vital for optimal sentiment classification. [ABSTRACT FROM AUTHOR]

INTRODUCTION: This paper highlights the potential of using generalized language models to extract structured texts from natural language descriptions of workflows in various industries like healthcare domain OBJECTIVES: Despite the criticality of these workflows to the business, they are often not fully automated or formally specified. Instead, employees may rely on natural language documents to describe the procedures. Text generation methods offer a way to extract structured plans from these natural language documents, which can then be used by an automated system. METHODS: This paper explores the effectiveness of using generalized language models, such as GPT-2, to perform text generation directly from these texts RESULTS: These models have already shown success in multiple text generation tasks, and the paper's initial results suggest that they could also be effective in text generation in healthcare domain. In fact, the paper demonstrates that GPT-2 can generate comparable results to many current text generation methods. CONCLUSION: This suggests that generalized language models can increase the efficiency and accuracy in text generation, where workflows are repetitive and sequential. [ABSTRACT FROM AUTHOR]

Methods for producing summaries from structured data have gained interest due to the huge volume of available data in the Web. Simultaneously, there have been advances in natural language generation from Resource Description Framework (RDF) data. However, no efforts have been made to generate natural language summaries for groups of multiple RDF entities. This paper describes the first algorithm for summarising the information of a set of RDF entities in the form of human-readable text. The paper also proposes an experimental design for the evaluation of the summaries in a human task context. Experiments were carried out comparing machine-made summaries and summaries written by humans, with and without the help of machine-made summaries. We develop criteria for evaluating the content and text quality of summaries of both types, as well as a function measuring the agreement between machine-made and human-written summaries. The experiments indicated that machine-made natural language summaries can substantially help humans in writing their own textual descriptions of entity sets within a limited time. [ABSTRACT FROM AUTHOR]

Large Language Models (LLMs) are being used for many language-based tasks, including translation, summarization and paraphrasing, sentiment analysis, and for content-generation tasks, such as code generation, answering search queries in natural language, and to power chatbots in customer service and other domains. Since much modern lexicography is based on investigation and analysis of large-scale corpora analogous to the (much larger) corpora used to train LLMs, we hypothesize that LLMs could be used for typical lexicographic tasks. A commercially-available LLM API (OpenAI’s ChatGPT gpt-3.5-turbo) was used to complete typical lexicographic tasks, such as headword expansion, phrase and form finding, and creation of definitions and examples. The results showed that the output of this LLM is not up to the standard of human editorial work, requiring significant oversight because of errors and “hallucinations” (the tendency of LLMs to invent facts). In addition, the externalities of LLM use, including concerns about environmental impact and replication of bias, add to the overall cost. [ABSTRACT FROM AUTHOR]

Large language models (LLMs) have significantly advanced natural language processing, but their progress has yet to be equal across languages. While most LLMs are trained in high-resource languages like English, multilingual models generally underperform monolingual ones. Additionally, aspects of their multilingual foundation sometimes restrict the byproducts they produce, like computational demands and licensing regimes. In this study, we document the development of open-foundation models tailored for use in low-resource settings, their limitations, and their benefits. This is the TeenyTinyLlama pair: two compact models for Brazilian Portuguese text generation. We release them under the permissive Apache 2.0 license on GitHub and Hugging Face for community use and further development.

Abstract In this study, we primarily aim to address the exposure bias issue in long text generation intrinsic to statistical language models. We propose a sentence-level heuristic tree search algorithm, specially tailored for long text generation, to mitigate the problem by managing generated texts in a tree structure and curbing the compounding of biases. Our algorithm utilizes two pre-trained language models, an auto-regressive model for generating new sentences and an auto-encoder model for evaluating sentence quality. These models work in tandem to perform four critical operations: expanding the text tree with new sentences, evaluating the quality of the additions, sampling potential unfinished text fragments for further generation, and pruning leaf nodes deemed unpromising. This iterative process continues until a pre-defined number of [EOS] tokens are produced, at which point we select the highest-scoring completed text as our final output. Moreover, we pioneer two novel token-level decoding techniques—nucleus sampling with temperature and diverse beam search with sampling. These methods, integrated with our sentence-level search algorithm, aim to improve the consistency and diversity of text generation. Experimental results, both automated measures (including Jaccard similarity, Word2vec similarity, and unique word ratio) and human evaluations (assessing consistency, fluency, and rhetorical skills), conclusively demonstrate that our approach considerably enhances the quality of machine-generated long-form text. Through this research, we aim to inspire further innovations in sentence-level search-based text generation algorithms.

The rise of advanced language models like GPT-3 and text generation has witnessed remarkable progress. However, leveraging the vast amount of knowledge within these models to enhance information retrieval performance remains an area that needs to be explored. This research used Artificial Intelligence, specifically the OpenAI GPT-3 language model, to create an application to help make written content. This research investigates the impact of incorporating GPT-3's knowledge into text generation processes and evaluates its influence on information retrieval tasks. Several features in text generation generate text that requires exact information, such as specifications for a product and accurate descriptions of a job or product, which are included in the concept of information retrieval in text creation by language models. The research used the few-shot learning method in the GPT-3 language model. The generated responses are then evaluated using established information retrieval metrics such as precision, recall, and F1-score. The findings of this research reveal the effectiveness of utilizing GPT-3's knowledge in enhancing information retrieval performance. The generated responses demonstrate improved relevance to user queries, resulting in the same performance precision and recall scores compared to other paid text generator websites. Application results are testing in capabilities of retrieving some information. Application capabilities tested on other commercial text generator engines. The test results obtained BERTscore 86\% (precision), 88\% (recall), and 87\% (F1-Score).

Large Language Models (LLMs), like ChatGPT, have sparked considerable interest among researchers across diverse disciplines owing to their remarkable text processing and generation capabilities. While ChatGPT is typically employed for tasks involving general knowledge, researchers increasingly explore the potential of this LLM-based tool in specific domains to enhance productivity. This study aims to compare the performance of a finetuned BERT model with that of ChatGPT on a domain-specific dataset in the context of developing an intelligent design support system. Through experiments conducted on classification and generation tasks, the knowledge transfer and elicitation abilities of ChatGPT are examined and contrasted with those of the finetuned BERT model. The findings indicate that ChatGPT exhibits comparable performance to the finetuned BERT model in sentence-level classification tasks but struggles with short sequences. However, ChatGPT's classification performance significantly improves when a few-shot setting is applied. Moreover, it can filter out unrelated data and enhance dataset quality by assimilating the underlying domain knowledge. Regarding content generation, ChatGPT with a zero-shot setting produces informative and readable output for domain-specific questions, albeit with an excessive amount of unrelated information, which can burden readers. In conclusion, ChatGPT demonstrates a promising potential for application in facilitating data labeling, knowledge transfer, and knowledge elicitation tasks. With minimal guidance, ChatGPT can substantially enhance the efficiency of domain experts in accomplishing their objectives. The findings suggest a nuanced integration of artificial intelligence (AI) with human expertise, bridging the gap from mere classification models to sophisticated human-analogous text generation systems. This signals a future in AI-augmented engineering design where the robust capabilities of AI technologies integrate with human creativity and innovation, creating a dynamic interactions to redefine how we tackle design challenges.

Diffusion models are a kind of math-based model that were first applied to image generation. Recently, they have drawn wide interest in natural language generation (NLG), a sub-field of natural language processing (NLP), due to their capability to generate varied and high-quality text outputs. In this article, we conduct a comprehensive survey on the application of diffusion models in text generation. We divide text generation into three parts (conditional, unconstrained, and multi-mode text generation, respectively) and provide a detailed introduction. In addition, considering that autoregressive-based pre-training models (PLMs) have recently dominated text generation, we conduct a detailed comparison between diffusion models and PLMs in multiple dimensions, highlighting their respective advantages and limitations. We believe that integrating PLMs into diffusion is a valuable research avenue. We also discuss current challenges faced by diffusion models in text generation and propose potential future research directions, such as improving sampling speed to address scalability issues and exploring multi-modal text generation. By providing a comprehensive analysis and outlook, this survey will serve as a valuable reference for researchers and practitioners interested in utilizing diffusion models for text generation tasks.

Sequential decision-making tasks that require information integration over extended durations of time are challenging for several reasons, including the problem of vanishing gradients, long training times and significant memory requirements. To this end, we propose a neuron model fashioned after the JK flip-flops in digital systems. A flip-flop is a sequential device that can store state information of the previous history. We incorporate the JK flip-flop neuron into several deep network architectures and apply the networks to difficult sequence processing problems. The proposed architectures include flip-flop neural networks (FFNNs), bidirectional flip-flop neural networks (BiFFNNs), convolutional flip-flop neural networks (ConvFFNNs), and bidirectional convolutional flip-flop neural networks (BiConvFFNNs). Learning rules of proposed architectures have also been derived. We have considered the most popular benchmark sequential tasks like signal generation, sentiment analysis, handwriting generation, text generation, video frame prediction, lung volume prediction, and action recognition to evaluate the proposed networks. Finally, we compare the results of our networks with the results from analogous networks with Long Short-Term Memory (LSTM) neurons on the same sequential tasks. Our results show that the JK flip-flop networks outperform the LSTM networks significantly or marginally on all the tasks, with only half of the trainable parameters. [ABSTRACT FROM AUTHOR]

Automatic Speech Recognition (ASR) is a crucial technology that enables machines to automatically recognize human voices based on audio signals. In recent years, there has been a rigorous growth in the development of ASR models with the emergence of new techniques and algorithms. One such model is the Whisper ASR model developed by OpenAI, which is based on a Transformer encoder-decoder architecture and can handle multiple tasks such as language identification, transcription, and translation. However, there are still limitations to the Whisper ASR model, such as speaker diarization, summarization, emotion detection, and performance with Indian regional languages like Hindi, Marathi and others. This research paper aims to enhance the performance of the Whisper ASR model by adding additional components or features such as speaker diarization, text summarization, emotion detection, text generation and question answering. Additionally, we aim to improve its performance in Indian regional languages by training the model on common voice 11 dataset from huggingface. The research findings have the potential to contribute to the development of more accurate and reliable ASR models, which could improve human-machine communication in various applications. [ABSTRACT FROM AUTHOR]

Creative writing is a challenge, and everyone faces at some point of time. The struggle of writing with the help of web is really high because of the fact that even though we try, we cannot really bypass the plagiarism. Hence, we created this tool named, Hailey - Text Generation Tool for Writing Assistance using Transformer. It leverages the power of GPT-2 Transformer and generate completion texts conditionally. This tool helps users by finishing their sentences without conditions. Users just have to enter their desired sentences and the tool will complete the paragraph by adding follow up sentences and words. This tool will be really helpful when users want to generate content in a short period of time or for getting new ideas for framing the article, stories and so on. [ABSTRACT FROM AUTHOR]

Participating in online communities has significant benefits to students learning in terms of students' motivation, persistence, and learning outcomes. However, maintaining and supporting online learning communities is very challenging and requires tremendous work. Automatic support is desirable in this situation. The purpose of this work is to explore the use of deep learning algorithms for automatic text generation in providing emotional and community support for a massive online learning community, Scratch. Particularly, state-of-art deep learning language models GPT-2 and recurrent neural network (RNN) are trained using two million comments from the online learning community. We then conduct both a readability test and human evaluation on the automatically generated results for offering support to the online students. The results show that the GPT-2 language model can provide timely and human-written like replies in a style genuine to the data set and context for offering related support. [ABSTRACT FROM AUTHOR]

SLR, which assists hearing-impaired people to communicate with other persons by sign language, is considered as a promising method. However, as the features of some of the static SL could be the same as the feature in a single frame of dynamic Isolated Sign Language (ISL), the generation of accurate text corresponding to the SL is necessary during the SLR. Therefore, Edge-directed Interpolation-based Recurrent Neural Network (EI-RNN)-centered text generation with varied features of the static and dynamic Isolated SL is proposed in this article. Primarily, ISL videos are converted to frames and pre-processed with key frame extraction and illumination control. After that, the foreground is separated with the Symmetric Normalised Laplacian-centered Otsu Thresholding (SLOT) technique for finding accurate key points in the human pose. The human pose’s key points are extracted with the Media Pipeline Holistic (MPH) pipeline approach and to improve the features of the face and hand sign, the resultant frame is fused with the depth image. After that, to differentiate the static and dynamic actions, the action change in the fused frames is determined with a correlation matrix. After that, to engender the output text for the respective SL, features are extracted individually as of the static and dynamic frames. It is obtained from the analysis that when analogized to the prevailing models, the proposed EI-RNN’s translation accuracy is elevated by 2.05% in INCLUDE 50 Indian SL based Dataset and Top 1 Accuracy 2.44% and Top 10 accuracy, 1.71% improved in WLASL 100 American SL. [ABSTRACT FROM AUTHOR]

Using computers to replace pilot seats in air traffic control (ATC) simulators is an effective way to improve controller training efficiency and reduce training costs. To achieve this, we propose a deep reinforcement learning model, RoBERTa-RL (RoBERTa with Reinforcement Learning), for generating pilot repetitions. RoBERTa-RL is based on the pre-trained language model RoBERTa and is optimized through transfer learning and reinforcement learning. Transfer learning is used to address the issue of scarce data in the ATC domain, while reinforcement learning algorithms are employed to optimize the RoBERTa model and overcome the limitations in model generalization caused by transfer learning. We selected a real-world area control dataset as the target task training and testing dataset, and a tower control dataset generated based on civil aviation radio land-air communication rules as the test dataset for evaluating model generalization. In terms of the ROUGE evaluation metrics, RoBERTa-RL achieved significant results on the area control dataset with ROUGE-1, ROUGE-2, and ROUGE-L scores of 0.9962, 0.992, and 0.996, respectively. On the tower control dataset, the scores were 0.982, 0.954, and 0.982, respectively. To overcome the limitations of ROUGE in this field, we conducted a detailed evaluation of the proposed model architecture using keyword-based evaluation criteria for the generated repetition instructions. This evaluation criterion calculates various keyword-based metrics based on the segmented results of the repetition instruction text. In the keyword-based evaluation criteria, the constructed model achieved an overall accuracy of 98.8% on the area control dataset and 81.8% on the tower control dataset. In terms of generalization, RoBERTa-RL improved accuracy by 56% compared to the model before improvement and achieved a 47.5% improvement compared to various comparative models. These results indicate that employing reinforcement learning strategies to enhance deep learning algorithms can effectively mitigate the issue of poor generalization in text generation tasks, and this approach holds promise for future application in other related domains. [ABSTRACT FROM AUTHOR]