Search

The appearance of Large Language Models (LLM) has implied a qualitative step forward in the performance of conversational agents, and even in the generation of creative texts. However, previous applications of these models in generating dialogues neglected the impact of 'hallucinations' in the context of generating synthetic dialogues, thus omitting this central aspect in their evaluations. For this reason, we propose an open-source and flexible framework called GenEvalGPT framework: a comprehensive multi-stage evaluation strategy utilizing diverse metrics. The objective is two-fold: first, the goal is to assess the extent to which synthetic dialogues between a chatbot and a human align with the specified commands, determining the successful creation of these dialogues based on the provided specifications; and second, to evaluate various aspects of emotional and subjective responses. Assuming that dialogues to be evaluated were synthetically produced from specific profiles, the first evaluation stage utilizes LLMs to reconstruct the original templates employed in dialogue creation. The success of this reconstruction is then assessed in a second stage using lexical and semantic objective metrics. On the other hand, crafting a chatbot's behaviors demands careful consideration to encompass a diverse range of interactions it is meant to engage in. Synthetic dialogues play a pivotal role in this context, as they can be deliberately synthesized to emulate various behaviors. This is precisely the objective of the third stage: evaluating whether the generated dialogues adhere to the required aspects concerning emotional and subjective responses. To validate the capabilities of the proposed framework, we applied it to recognize whether the chatbot exhibited one of two distinct behaviors in the synthetically generated dialogues: being emotional and providing subjective responses, or remaining neutral. This evaluation will encompass traditional metrics and automatic metrics generated by the LLM. In our use case of art-related dialogues, our findings reveal that the capacity to recover templates or profiles is more effective for information or profile items that are objective and factual, in contrast to those related to mental states or subjective facts. For the emotional and subjective behavior assessment, rule-based metrics achieved a 79% of accuracy in detecting emotions or subjectivity (anthropic), and an 82% on the LLM automatic metrics. The combination of these metrics and stages could help to decide which of the generated dialogues should be maintained depending on the applied policy, which could vary from preserving between 57% to 93% of the initial dialogues. • Large Language Models (LLM) can generate and evaluate novel dialogues. • A combination of objective metrics with LLM self-reported metrics helps to assess quality. • The proposed framework allows the generation and evaluation of synthetic dialogues. • Framework's use case: emotional and subjective characteristics of chatbots (anthropic). [ABSTRACT FROM AUTHOR]

In this paper, we propose a robust methodology to assess the value of microblogging data to forecast stock market variables: returns, volatility and trading volume of diverse indices and portfolios. The methodology uses sentiment and attention indicators extracted from microblogs (a large Twitter dataset is adopted) and survey indices (AAII and II, USMC and Sentix), diverse forms to daily aggregate these indicators, usage of a Kalman Filter to merge microblog and survey sources, a realistic rolling windows evaluation, several Machine Learning methods and the Diebold-Mariano test to validate if the sentiment and attention based predictions are valuable when compared with an autoregressive baseline. We found that Twitter sentiment and posting volume were relevant for the forecasting of returns of S&P 500 index, portfolios of lower market capitalization and some industries. Additionally, KF sentiment was informative for the forecasting of returns. Moreover, Twitter and KF sentiment indicators were useful for the prediction of some survey sentiment indicators. These results confirm the usefulness of microblogging data for financial expert systems, allowing to predict stock market behavior and providing a valuable alternative for existing survey measures with advantages (e.g., fast and cheap creation, daily frequency). [ABSTRACT FROM AUTHOR]

GUI is an essential factor influencing users' perception of an app. As the visual bridge between the user and the application, the quality of a GUI should be measured by the users' subjective feelings. Therefore, to improve the UI, developers should pay attention to what users think and actively respond to their feedback. User reviews from different channels provide a way for developers to obtain users' suggestions about the UI, and it could be of great benefit to improve the UI of their apps. Methods to mine different information (e.g., bug report, feature request) from reviews have been proposed in the previous works, however, suggestion information for supporting UI improvement has not gained much attention. In this paper, we propose UISMiner (UI Suggestion Miner), a novel approach to automatically mine the suggestions about the UI design from reviews. The application of UISMiner relies on natural language processing and machine learning classifier, which involves three main steps: 1) gaining reviews related to UI suggestion by training a classifier; 2) extracting UI suggestions from the obtained reviews by defining rules. 3) associating UI suggestions with the corresponding part in the UI. The experiment based on Google Play shows that UISMiner can effectively obtain reviews related to UI suggestions and extract such information from the gained reviews, the F-measure can reach 87.71% and 76.99% respectively. Furthermore, the survey on 10 developers shows that the information provided by us is useful for updating the UI of apps. [ABSTRACT FROM AUTHOR]

This study aims to systematically review the cross disciplinary literature covering the time period from 1934 to January 2013 on behavioral operations in supply chain in order to identify and define the taxonomy of the research on power influences in supply chain. A list of noted journals and search results from Science Direct and Web of Knowledge, IEEE Xplore, and INFORMS (approximately 11,000 journal articles) is used to prepare content collection. Latent Semantic Analysis (LSA) is applied as the review and knowledge extraction methodology. Using the text analysis and mining method we can combine statistical methods and expert human judgment to extract knowledge in the form of key latent factors. The LSA based analysis gives the study a scientific grounding which helps to overcome the subjectivity of collective opinion about the trends. This approach allows proposing taxonomy of the research on power influences in supply chain. The adopted systems approach is used to find research gaps in each class of taxonomy. An emerging trend is noticed in the research of behavioral operations in supply chain. Understanding such a scholarly structure and future trends will assist researchers to assimilate the divergent developments of this multidisciplinary research in one place. This review will be beneficial for practitioners as they consider behavioral aspects in decision making. We have also studied articles related to supply chain published in Expert Systems with Applications (ESWA) journal. We have speculated what an ESWA-related community would like to see in future publications. This will encourage researchers to explore the recommended areas and publish to these outlets. [ABSTRACT FROM AUTHOR]

• Time burst characteristic is studied by 3 aspects in fake review identification. • A suspicion degree method of fake reviews is proposed from (1) by LOF algorithm. • A fake review identification framework is given including the suspicion degree. • The case study shows that the proposed method outperforms the existed methods. With the rapid development of e-commerce, online reviews have played an increasingly important role in consumers' shopping intentions and behaviors. Therefore, how to effectively identify fake reviews has become one of the important issues that need to be resolved. Since the existing methods do not fully consider the time burst characteristics of reviews, this paper proposes a suspicion degree determining method based on the three-dimensional time series. Besides, combining the suspicion degree feature, review text features, and reviewer's behavior features together, this paper proposes a more comprehensive fake review identification framework. The yelp and amazon public data sets are carried out to verify the effectiveness of the proposed method, and the experimental results show that the proposed method outperforms the most advanced methods. [ABSTRACT FROM AUTHOR]