Search

Security and performance are critical nonfunctional requirements for software systems. Thus, it is crucial to include verification activities during software development to identify defects related to such requirements, avoiding their occurrence after release. Software verification, including testing and reviews, encompasses a set of activities that have a purpose of analyzing the software searching for defects. Security and performance verification are activities that look at defects related to these specific quality attributes. Few empirical studies have been focused on how is the state of the practice in security and performance verification. This paper presents the results of a case study performed in the context of Brazilian organizations aiming to characterize security and performance verification practices. Additionally, it provides a set of conjectures indicating recommendations to improve security and performance verification activities.

Background: research synthesis is still challenge in Software Engineering due to the heterogeneity of primary studies in the area. Also, it generates a significant volume of information which is complex to manage. Aims: to provide support to this kind of studies in SE. Method: we present the Evidence Factory, a tool designed to support the Structured Synthesis Method (SSM). SSM is a research synthesis method that can be used to aggregate both quantitative and qualitative studies. It is a kind of integrative synthesis method, such as meta-analysis, but has several features from interpretative methods, such as meta-ethnography, particularly those concerned with conceptual development. Results: the tool is a web-based infrastructure, which supports the organization of synthesis studies. Researchers can compare findings from different studies by modeling their results according to the evidence meta-model. After deciding whether the evidence can be combined, the tool automatically computes the uncertainty associated with the aggregated results using the formalisms from the Mathematical Theory of Evidence. Conclusion: the tool was used in real synthesis studies and is freely available for the SE community.

Context: Surveys constitute an valuable tool to capture a large-scale snapshot of the state of the practice. Apparently trivial to adopt, surveys hide, however, several pitfalls that might hinder rendering the result valid and, thus, useful. Goal: We aim at providing an overview of main pitfalls in software engineering surveys and report on practical ways to deal with them. Method: We build on the experiences we collected in conducting many studies and distill the main lessons learnt. Results: The eight lessons learnt we report cover different aspects of the survey process ranging from the design of initial research objectives to the design of a questionnaire. Conclusions: Our hope is that by sharing our lessons learnt, combined with a disciplined application of the general survey theory, we contribute to improving the quality of the research results achievable by employing software engineering surveys., Final CR version

Context: The results of large-scale studies in software engineering can be significantly impacted by samples' representativeness. Diverse population sources can be used to support sampling for such studies. Goal: To compare two samples, one from the crowdsourcing platform Mechanical Turk and another from the professional social network LinkedIn, in an online experiment for evaluating the relevance of Java code snippets to programming tasks. Method: To compare the samples (subjects' experience, programming habits) and experimental results concerned with three experimental trials. Results: LinkedIn's subjects present significantly higher levels of experience in Java programming and programming in general than Mechanical Turk's subjects. The experimental results revealed a significant difference between samples and suggested that LinkedIn's subjects were more pessimistic than Mechanical Turk's subjects despite a high level consistency in the experimental results. Conclusion: The combined use of sources of sampling can bring benefits to large scale studies in software engineering, especially when heterogeneity is desired in the population. Thus, it can be useful to investigate and characterize alternative sources of sampling for performing large-scale studies in software engineering.

Context: Several empirical studies investigated the benefits and drawbacks of acquiring a Software Reference Architecture (SRA) to construct a family of software systems with similar architectural needs. However, these empirical results have not been synthesized by any study yet. Such synthesized evidence is essential to make informed decisions whether or not to adopt an SRA in an organization. Goal: To aggregate existing empirically- grounded evidence about the benefits and drawbacks of SRAs, aiming at supporting organizations' decision making on their adoption. Method: To identify primary studies in the technical literature through a systematic literature review, and then, use the Structured Synthesis Method (SSM) to aggregate qualitative and quantitative evidence through the use of diagrammatic models. Results: From the five identified primary studies, five SRA benefits have considerably increased their belief value after aggregation: interoperability of software systems, reduced development costs, improved communication among stakeholders, reduced risk, and reduced time- to-market. Also, one drawback of SRAs has increased its belief value: the required learning curve for developers. Conclusions: The aggregated results consolidate knowledge and confidence on some of the studied SRA effects. The commonly reported effects showed a clear increment of their belief and pointed out to broader generalization. The effects that did not show any belief increment are important to detect areas requiring further evidence to reach a higher degree of consolidation. Practitioners might benefit from these results to support the decision of adopting an SRA in practice.

All software system fails. However, as more defects the development team can identify and remove as lower will be the probability of software to fail. Identifying and removing defects throughout the software development process represent an important approach to improve software quality. There is evidence about the feasibility, efficiency and efficacy of using software inspections to support the identification of defects. Besides their low cost, the performance of inspections can be positively influenced by the technique used to detect defects. Despite all these benefits, the use of inspections is not common place in the software industry. Therefore, this Quatic 2014 keynote speech intends to discuss the benefits of Software Inspections (the method and techniques) and their use in software projects.

Quantitative studies in Software Engineering are frequently dependent on primary studies in which population is usually small and established by convenience. It brings several limitations for the analysis and strength of results due sampling issues. Therefore, when these studies are reapplied, different and non-clustered populations are established, making unfeasible evidence generalization and contributing for an imbalance between research and practice. Aiming at investigating ways to overcome the absence of large sampling frames in Software Engineering studies, this short paper presents the results of an initial experience concerned with the systematic recruitment of subjects for a survey regarding software requirements effort factors by using social networks compared with recruitment by convenience. We have observed in this particular case that using social networks technology does not guarantee sample enlargement by just posting invitations in specific forums. However, its usage can contribute to increase the subjects' heterogeneity and to increase the level of confidence of the sample, which consequently improve our capacity of observing the object under study, with the probable strengthen of results.

Usability is one of the most important software qualities regarding its acceptability by end users. It is even more important in the context of Web applications. One way of evaluating applications usability is through inspections. The WDP (Web Design Perspectives-based Usability Evaluation) is a usability inspection technique presenting evidence of industrial use feasibility; however the need for some computerized support was suggested by its users. Therefore, the WDP Tool was built aiming to provide automated support to the application of the WDP technique, reducing the effort involved in usability inspections with WDP. This paper presents two observational studies regarding the use of the WDP Tool, which aimed to analyze the cost-effectiveness of its application and its appropriateness to the industrial environment through the Technology Acceptance Model (TAM). The results provide feasibility indication of using the WDP Tool to support usability inspections in real software development projects in industry.

Over the past years Brazil has done big investments in software quality. In 2010 the investment was among the highest in the world. Among them, the long term MPS. BR program focuses on improving the software process quality of Brazilian companies. Therefore, this paper intends to describe its software process improvement reference model (MR-MPS) and the most important results accomplished to date on Government, Industry and University scenarios. The Brazilian Government had endorsed the MPS adoption as a mean to increase the software industry competitiveness both locally and globally. In the University scenario, the MPS model contributes to transfer software engineering knowledge to the Industry and allows the identification of new research challenges. The quantification of benefits in the Industry is based on the iMPS family of surveys which allow both performance results characterization and performance variation analysis considering organizations that adopted the MPS model. Quantitative data have shown the MPS model acceptance by the Industry. Moreover, the obtained results evidence improvements reduction of cost and schedule and increase of productivity and quality in the software projects.

The higher the quality of the specification document, the lower the effort of its translation into design models and testing plans. Besides, an adequate level of abstraction to promote such translations must be described. Therefore, to ensure the quality of requirements specifications it is strategic to develop high quality software applications. So, in this paper a model-based approach to support the correctness, structuring, and translation of functional requirements specifications is described. This approach consists of facilities to build and inspect requirements specifications based on activity diagrams (capturing use cases), and derive functional tests from them. A tool to model and check the activity diagram, a checklist-based inspection technique and a test procedure generation tool form it. This approach was assessed in experimental studies that indicated its feasibility in specification time and a significant reduction of defects in the specified use cases when compared to ad-hoc approaches.

Software Product Line Engineering aims to ensure the correctness, completeness and consistency among its artifacts and the specified domain, in order to prevent the spread of defects for the products derived from this domain. Among initial artifacts of a software product line, feature models are artifacts generated in various domain engineering approaches. Although software inspection is highlighted as an effective review activity for detection of defects in software artifacts, mainly in the early models of software projects, a recent quasi-systematic review of literature indicated a lack of techniques to support the inspection of software product line artifacts, which include features models. Thus, this paper presents FMCheck, a checklist-based inspection technique to support the detection of defects on feature models. This technique was developed to be configurable and to be applied on several extensions of the original feature model notation presented by FODA approach, including the Odyssey-FEX notation, in particular. FMCheck was submitted to a proof of concept and to a further in vitro feasibility study, in which it could be seen the feasibility of FMCheck application and also that inspections applying FMCheck are more effective than ad-hoc inspections, when applied on four distinct domains.

Despite the importance of assure quality of Web applications for their use by the contemporary society, just a few Web development methods offer mechanisms to support such quality in the final product. Besides, when quality concern is present, it is usually concerned with the design models and implementation issues. With some risk, all of them assume the requirements specification is ready and good enough to effectively support the design models generation and consequent implementation. However, the quality of the requirements can affect software design and implementation. Therefore, this paper investigates an approach to support the specification and quality assurance of functional requirements concerned with Web applications. This approach makes use of stereotyped UML activity diagrams to support functional requirements specification. A use case tool and a checklist based inspection technique complement it. The results of two experimental studies indicated its feasibility on supporting the specification of requirements for real web applications. It has been observed through the reduction of the specification time and number of defects in activity diagrams (and their corresponding use cases) specified for an industrial large scale Web-based information system when compared with ad-hoc specification approaches.

Objective: to investigate studies about software processes looking for practices which can be used to obtain agility in software processes. Method: A systematic review including seven search engines was executed in Feb/2010. To apply the defined criteria to select papers and extract information regarding working practices bringing agility to software processes. Results: from 6696 retrieved papers, 441 were selected to support the identification of 236 occurrences of 51 distinct practices associated with the concept of agility. Their descriptions were deeply analyzed and consolidated. After discarding those which appeared in the technical literature in a small amount of papers, 17 agile practices were identified. Conclusion: although further studies are necessary to evaluate the efficacy of these 17 agile practices, 12 of them have been more commonly approached in the software projects and could be primarily considered: test driven development, continuous integration, pair programming, planning game, onsite customer, collective code ownership, small releases, metaphor, refactoring, sustainable pace, simple design and coding standards.

The combination of testing techniques is considered an effective strategy to evaluate a software product. However, the selection of which techniques to combine in a software project has been an interesting challenge in the Software Engineering field. This paper presents a proposal extending an approach developed to support the combined selection of model-based testing (MBT) techniques, named Porantim, applying Multiobjective Combinatorial Optimization strategy by determining the smallest dominating set in a bipartite and weighted graph. Thus, a local search strategy algorithm is proposed generating solutions aiming at maximizing the coverage of software project characteristics and skills required by the testing team to use the techniques and minimizing the eventual effort to construct models used for test cases generation. A preliminary evaluation analyzes this new approach when compared to the Porantim's original version, and the results indicate improvements in the MBT techniques selection.

The requirements specification of contemporary software applications usually is composed by diverse artifacts describing lots of activities, flows, dependencies among the flows, branches and business rules. For instance, web or scientific workflow (e-science) based applications require structural representations for the various activities and functionalities involved in their execution, usually described trough activity diagrams. The quality assurance of such specifications represents a challenge for software engineers. The results of a quasi-systematic review indicated there is a lack of software technologies to support the inspection of this type of requirements specification. Therefore, in this paper, besides the review results, an inspection technique (checklist) to review Activity Diagrams on the requirements specifications is introduced. A proof of concept on applying the checklist for the inspection of a real requirements specification concerned with a scientific workflow based application indicated some advantages on its defect detection capacity when compared with previously executed ad-hoc inspection by software engineers.

In December 2003 a Brazilian nationwide program was created aiming at improving software processes. The MPS.BR Program is responsible for a great evolution in the software development scenario in Brazil by providing the means to support software process improvement initiatives based on the MPS Model. In 2008 a project named iMPS was launched to enable the analysis of the Performance Results of organizations that adopted the MPS Model. This paper describes some of the findings of the studies that have already been conducted. Preliminary evidences show improvement trends regarding cost, quality, schedule, and productivity, especially for those organizations that adopted the MPS Model for continuous improvement purposes.

Experimental studies execution in Software Engineering is time consuming and produces great amount of information and knowledge with complex management. Hence, it is important to have a computerized infrastructure to support the experimental study execution. This infrastructure should provide efficient knowledge representation and management mechanisms regarding Experimentation. In this paper we discuss the knowledge repository structure of an experimentation environment in Software Engineering (eSEE) – an infrastructure capable of instantiating software engineering environments to manage knowledge about the definition, planning, execution and packaging of experimental studies in software engineering.

This paper presents a literature survey of action research (AR) studies published in nine major Software Engineering (SE) journals and three conference proceedings in the period 1993 to June 2009. A strict selection based on distinguishing SE from Information Systems research has identified 16 papers. Although they represent a very small fraction of the studies being conducted in SE, such papers concern with different SE contexts allowing to get information about the increasing tendency in the AR use in software engineering. However, as shown by the initial results, SE researchers should invest more on rigor when defining, applying and reporting AR studies inSE.

The selection of software technologies represents a risk factor to a software project. Therefore, using tailored software technologies to support this task can contribute to reduce the risk of inadequate choices made by software engineers. This paper presents the results of an experimental study conducted to evaluate if three dependent variables (selection time, % of correct choices, and used information) can be significantly affected by two different approaches to support the selection of Model-Based Testing techniques. This study consists of an external replication of an experiment conducted to evaluate two approaches for the selection of testing techniques. Therefore, we present the adaptations performed in the planning and design of the experiment and the challenges observed in conducting a study replication. The results indicated that the time spent to select MBT techniques and the percentage of right selections can be affected by the approach used to select MBT techniques for different software project categories.

We have proposed a checklist-based usability inspection technique (WDP - Web Design Perspectives-Based Usability Evaluation) specific for Web applications’ usability evaluation and we are following an experimentation-based methodology to support its development and improvement. This paper describes an observational study conducted, aimed at eliciting how inspectors apply the WDP technique. We discuss the quantitative and qualitative results of the result and their impact on improving the WDP technique. We analyzed the qualitative data using the procedures from the Grounded Theory (GT) method.

The technical literature regarding Model-based Testing (MBT) has several techniques with different characteristics and goals available to be applied in software projects. Besides the lack of information regarding these techniques, they could be applied together in a software project aiming at improving the testing coverage. However, this decision needs to be carefully analyzed to avoid loss of resources in a software project. Based on this scenario, this paper proposes an approach with the purpose of supporting the unique or combined selection of MBT techniques for a given software project considering two aspects: the adequacy level between MBT techniques and the software project characteristics and impact of more than one MBT technique in some testing process variables. At the end, preliminary results of an experimental evaluation are presented.

Defect causal analysis (DCA) is a means of product focused software process improvement. A systematic literature review to identify the DCA state of the art has been undertaken. The systematic review gathered unbiased knowledge and evidence and identified opportunities for further investigation. Moreover, some guidance on how to efficiently implement DCA in software organizations could be elaborated. This paper describes the initial concept of the DBPI (Defect Based Process Improvement) approach. It represents a DCA based approach for process improvement, designed considering the results of the systematic review and the obtained guidance. Its main contributions are tailoring support for DCA based process improvement and addressing an identified opportunity for further investigation by integrating organizational learning mechanisms regarding cause-effect relations into the conduct of DCA.

Experimental replications are very important to the advancement of empirical software engineering. Replications are one of the key mechanisms to confirm previous experimental findings. They are also used to transfer experimental knowledge, to train people, and to expand a base of experimental evidence. Unfortunately, experimental replications are difficult endeavors. It is not easy to transfer experimental know-how and experimental findings. Based on our experience, this paper discusses this problem and proposes a Framework for Improving the Replication of Experiments (FIRE). The FIRE addresses knowledge sharing issues both at the intra-group (internal replications) and inter-group (external replications) levels. It encourages coordination of replications in order to facilitate knowledge transfer for lower cost, higher quality replications and more generalizable results.

Given the growth in the number and size of Web Applications worldwide, Web quality assurance, and more specifically Web usability have become key success factors. Therefore, this work proposes a usability evaluation technique based on the combination of Web design perspectives adapted from existing literature, and heuristics. This new technique is assessed using a controlled experiment aimed at measuring the efficiency and effectiveness of our technique, in comparison to Nielsen's heuristic evaluation. Results indicated that our technique was significantly more effective than and as efficient as Nielsen's heuristic evaluation.