Search

Lexical-based metrics such as BLEU, NIST, and WER have been widely used in machine translation MT evaluation. However, these metrics badly represent semantic relationships and impose strict identity matching, leading to moderate correlation with human judgments. In this paper, we propose a novel MT automatic evaluation metric Semantic Travel Distance STD based on word embeddings. STD incorporates both semantic and lexical features word embeddings and n-gram and word order into one metric. It measures the semantic distance between the hypothesis and reference by calculating the minimum cumulative cost that the embedded n-grams of the hypothesis need to “travel” to reach the embedded n-grams of the reference. Experiment results show that STD has a better and more robust performance than a range of state-of-the-art metrics for both the segment-level and system-level evaluation.

Large-scale online systems are complex, fast-evolving, and hardly bug-free despite the testing efforts. Backend system monitoring cannot detect many types of issues, such as UI related bugs, bugs with small impact on backend system indicators, or errors from third-party co-operating systems, etc. However, users are good informers of such issues: They will provide their feedback for any types of issues. This experience paper discusses our design of iFeedback, a tool to perform real-time issue detection based on user feedback texts. Unlike traditional approaches that analyze user feedback with computation-intensive natural language processing algorithms, iFeedback is focusing on fast issue detection, which can serve as a system life-condition monitor. In particular, iFeedback extracts word combination-based indicators from feedback texts. This allows iFeedback to perform fast system anomaly detection with sophisticated machine learning algorithms. iFeedback then further summarizes the texts with an aim to effectively present the anomaly to the developers for root cause analysis. We present our representative experiences in successfully applying iFeedback in tens of large-scale production online service systems in ten months.

Despite getting widely adopted recently, a Neural Machine Translation (NMT) system is often found to produce translation failures in the outputs. Developers have been relying on in-house system testing for quality assurance of NMT. This testing methodology requires human-constructed reference translations as the ground truth (test oracle) for example natural language inputs. The testing methodology has shown benefits of quickly enhancing an NMT system in early development stages. However, in industrial settings, it is desirable to detect translation failures without reliance on reference translations for enabling further improvements on translation quality in both industrial development and production environments. Aiming for a practical and scalable solution to such demand in the industrial settings, in this paper, we propose a new approach for automatically identifying translation failures without requiring reference translations for a translation task. Our approach focuses on a property of natural language translation that can be checked systematically by using information from both the test inputs (i.e., the texts to be translated) and the test outputs (i.e., the translations under inspection) of the NMT system. Our evaluation conducted on real-world datasets shows that our approach can effectively detect property violations as translation failures. By deploying our approach in the translation service of WeChat (a messenger app with more than one billion monthly active users), we show that our approach is both practical and scalable in the industrial settings.

It is vital for popular mobile apps with large numbers of users to release updates with rich features while keeping stable user experience. Timely and accurately locating emerging app issues can greatly help developers to maintain and update apps. User feedback (i.e., user reviews) is a crucial channel between app developers and users, delivering a stream of information about bugs and features that concern users. Methods to identify emerging issues based on user feedback have been proposed in the literature, however, their applicability in industry has not been explored. We apply the recent method IDEA to WeChat, a popular messenger app with over 1 billion monthly active users, and find that the emerging issues detected by IDEA are not stable (i.e., due to its inherent randomness, its results change when run multiple times even for the same inputs), and there are other problems such as long running time. To address these limitations, we design a novel tool, named DIVER. Different from IDEA, DIVER is more efficient (it can report real-time alerts in seconds), generates reliable results, and most importantly, achieves higher accuracy in our practice. After its deployment on WeChat, DIVER successfully detected 18 emerging issues of WeChat's Android and iOS apps in one month. Additionally, DIVER significantly outperforms IDEA by 29.4% in precision and 32.5% in recall.

Neural Machine Translation (NMT) has been widely adopted recently due to its advantages compared with the traditional Statistical Machine Translation (SMT). However, an NMT system still often produces translation failures due to the complexity of natural language and sophistication in designing neural networks. While in-house black-box system testing based on reference translations (i.e., examples of valid translations) has been a common practice for NMT quality assurance, an increasingly critical industrial practice, named in-vivo testing, exposes unseen types or instances of translation failures when real users are using a deployed industrial NMT system. To fill the gap of lacking test oracle for in-vivo testing of an NMT system, in this paper, we propose a new approach for automatically identifying translation failures, without requiring reference translations for a translation task; our approach can directly serve as a test oracle for in-vivo testing. Our approach focuses on properties of natural language translation that can be checked systematically and uses information from both the test inputs (i.e., the texts to be translated) and the test outputs (i.e., the translations under inspection) of the NMT system. Our evaluation conducted on real-world datasets shows that our approach can effectively detect targeted property violations as translation failures. Our experiences on deploying our approach in both production and development environments of WeChat (a messenger app with over one billion monthly active users) demonstrate high effectiveness of our approach along with high industry impact., Comment: 10 pages

Monkey, a random testing tool from Google, has been popularly used in industrial practices for automatic test input generation for Android due to its applicability to a variety of application settings, e.g., ease of use and compatibility with different Android platforms. Recently, Monkey has been under the spotlight of the research community: recent studies found out that none of the studied tools from the academia were actually better than Monkey when applied on a set of open source Android apps. Our recent efforts performed the first case study of applying Monkey on WeChat, a popular messenger app with over 800 million monthly active users, and revealed many limitations of Monkey along with developing our improved approach to alleviate some of these limitations. In this paper, we explore two optimization techniques to improve the effectiveness and efficiency of our previous approach. We also conduct manual categorization of not-covered activities and two automatic coverage-analysis techniques to provide insightful information about the not-covered code entities. Lastly, we present findings of our empirical studies of conducting automatic random testing on WeChat with the preceding techniques.

Given the ever increasing number of research tools to automatically generate inputs to test Android applications (or simply apps), researchers recently asked the question "Are we there yet?" (in terms of the practicality of the tools). By conducting an empirical study of the various tools, the researchers found that Monkey (the most widely used tool of this category in industrial practices) outperformed all of the research tools that they studied. In this paper, we present two significant extensions of that study. First, we conduct the first industrial case study of applying Monkey against WeChat, a popular messenger app with over 762 million monthly active users, and report the empirical findings on Monkey's limitations in an industrial setting. Second, we develop a new approach to address major limitations of Monkey and accomplish substantial code-coverage improvements over Monkey, along with empirical insights for future enhancements to both Monkey and our approach.

This paper conducts a formal study of the shot boundary detection problem. First, a general formal framework of shot boundary detection techniques is proposed. Three critical techniques, i.e., the representation of visual content, the construction of continuity signal and the classification of continuity values, are identified and formulated in the perspective of pattern recognition. Meanwhile, the major challenges to the framework are identified. Second, a comprehensive review of the existing approaches is conducted. The representative approaches are categorized and compared according to their roles in the formal framework. Based on the comparison of the existing approaches, optimal criteria for each module of the framework are discussed, which will provide practical guide for developing novel methods. Third, with all the above issues considered, we present a unified shot boundary detection system based on graph partition model. Extensive experiments are carried out on the platform of TRECVID. The experiments not only verify the optimal criteria discussed above, but also show that the proposed approach is among the best in the evaluation of TRECVID 2005. Finally, we conclude the paper and present some further discussions on what shot boundary detection can learn from other related fields

Web search engines have major impact in people's everyday life. It is of great importance to test the retrieval effectiveness of search engines. However, it is labor-intensive to judge the relevance of search results for a large number of queries, and these relevance judgments may not be reusable since the Web data change all the time. In this work, we propose to mine test oracles of Web search engines from existing search results. The main idea is to mine implicit relationships between queries and search results, e.g., some queries may have fixed top 1 result while some may not, and some Web domains may appear together in top 10 results. We define a set of items of queries and search results, and mine frequent association rules between these items as test oracles. Experiments on major search engines show that our approach mines many high-confidence rules that help understand search engines and detect suspicious search results.

Software systems are often built upon third party libraries. Developers may replace an old library with a new library, for the consideration of functionality, performance, security, and so on. It is tedious to learn the often complex APIs in the new library from the scratch. Instead, developers may identify the suitable APIs in the old library, and then find counterparts of these APIs in the new library. However, there is typically no such cross-references for APIs in different libraries. Previous work on automatic API recommendation often recommends related APIs in the same library. In this paper, we propose to mine search results of Web search engines to recommend related APIs of different libraries. In particular, we use Web search engines to collect relevant Web search results of a given API in the old library, and then recommend API candidates in the new library that are frequently appeared in the Web search results. Preliminary results of generating related C# APIs for the APIs in JDK show the feasibility of our approach.

A key component of automated object-oriented unit-test generation is to find method-call sequences that generate desired inputs of a method under test (MUT). Previous work cannot find desired sequences effectively due to the large search space of possible sequences. To address this issue, we present a MUT-aware sequence recommendation approach called RecGen to improve the effectiveness of random object-oriented unit-test generation. Unlike existing random testing approaches that select sequences without considering how a MUT may use inputs generated from sequences, RecGen analyzes object fields accessed by a MUT and recommends a short sequence that mutates these fields. In addition, for MUTs whose test generation keeps failing, RecGen recommends a set of sequences to cover all the methods that mutate object fields accessed by the MUT. This technique further improves the chance of generating desired inputs. We have implemented RecGen and evaluated it on three libraries. Evaluation results show that RecGen improves code coverage over previous random testing tools.

It is labor-intensive to manually verify the outputs of a large set of tests that are not equipped with test oracles. Test selection helps to reduce this cost by selecting a small subset of tests that are likely to reveal faults. A promising approach is to dynamically mine operational models as potential test oracles and then select tests that violate them. Existing work mines operational models from verified passing tests based on dynamic invariant detection. In this paper, we propose to mine common operational models, which are not always true in all observed traces, from a set of unverified tests based on mining predicate rules. Specifically, we collect values of simple predicates at runtime and then generate and evaluate predicate rules as potential operational models after running all the tests. We then select tests that violate the mined predicate rules for result inspection. Preliminary results on the Siemens suite and the grep program show the effectiveness of our approach.

We propose an incentive routing and forwarding scheme that integrates a reputation system into a monetary payment mechanism to encourage nodes cooperation in wireless ad hoc networks. For the first time in the literature, we build our reputation system based on a heat diffusion model. The heat diffusion model provides us a way of combining the direct and indirect reputation together and propagating the reputation from locally to globally. Further, we model and analyze our incentive scheme using a coalitional game, which is not the usual non-cooperative game like others. We further prove that under a proper condition this game has a non-empty stable core. From the evaluation we can see that the cumulative utility of nodes increases when nodes stay in the core.

In the paper, our video retrieval system is presented. The system acts as a decision support system to help users to find what they want with many analysis and visualization tools provided by the system. It consists of three basic retrieval models which searches shots in text, image and concept space respectively. The results from different modalities are fused to achieve better performance. The relevance shots are shown to users in different threads and expanded in different ways to help users try their best to make correct decision during the retrieval procedure.

Extraction and utilization of high-level semantic features are critical for more effective video retrieval. However, the performance of video retrieval hasn't benefited much despite of the advances in high-level feature extraction. To make good use of high-level semantic features in video retrieval, we present a method called pointwise mutual information weighted scheme(PMIWS). The method makes a good judgment of the relevance of all the semantic features to the queries, taking the characteristics of semantic features into account. The method can also be extended for the fusion of multi-modalities. Experiment results based on TRECVID2005 corpus demonstrate the effectiveness of the method.

Shot boundary detection servers as a preliminary step to structure the content of videos. Up to now, a large number of methods have been proposed. We give a brief overview of previous works with a novel view, focusing on the solutions of the two main disturbances, i.e., abrupt illuminance change and great camera or object motion. Then this paper presents a novel shot boundary detection framework, consisting of three components: fade out/in (abbreviated as FOI) detector, cut detector and gradual transition (abbreviated as GT) detector. The key technique of FOI detector is the recognition of monochrome frames. For cut detection, a second-order difference method is firstly applied to obtain candidate cuts, and then a post-processing procedure is taken to eliminate the false positives. In GT detector, the twin-comparison approach is employed to detect short gradual transition which lasts less than six frames, while for long gradual transition, an improvement of twin-comparison algorithm is designed. Firstly, to effectively reduce the false alarms of quick motion, the lower threshold is self-adaptive to motion feature. Secondly, an FSA (finite state automata) model is adopted to replace the twin-comparison strategy. This framework makes good use of various features and successfully integrates all the modules together. Finally, the system is evaluated on the TRECVID benchmarking platform and the experimental results reveal the effectiveness of our system.