Search

Stream processing has been an active research field for more than 20 years, but it is now witnessing its prime time due to recent successful efforts by the research community and numerous worldwide open-source communities. This survey provides a comprehensive overview of fundamental aspects of stream processing systems and their evolution in the functional areas of out-of-order data management, state management, fault tolerance, high availability, load management, elasticity, and reconfiguration. We review noteworthy past research findings, outline the similarities and differences between the first ('00–'10) and second ('11–'23) generation of stream processing systems, and discuss future trends and open problems. [ABSTRACT FROM AUTHOR]

Memory operations are critical to an application's reliability and performance. To reason about their correctness and track opportunities for optimisations, sophisticated instrumentation frameworks, such as Valgrind and Pin, have been developed. Both provide only limited facilities for analysing the collected data. This work presents a Valgrind's extension for examining a software applications' dynamic memory profile through live interactive analysis with SQL. The Pico COllections Query Library (pico ql) module maps Valgrind's data structures that contain the instrumented application's memory operations metadata to a relational interface. Queries are type‐safe and the module imposes only a trivial overhead when idle. The authors evaluate the proposed approach on ten applications and through a qualitative study. They find 900 kb of undefined bytes in bzip2 that account for 12% of its total memory use and a performance‐critical code execution path in the Unix commands sort and uniq. The referenced functions are part of glibc and have been independently modified to boost the library's performance. The qualitative study has users rate the usefulness, usability, effort, correctness, and expressiveness of pico ql queries compared to Python scripts. The findings indicate that querying with pico ql incurs lower user effort. [ABSTRACT FROM AUTHOR]

Memory operations are critical to an application's reliability and performance. To reason about their correctness and track opportunities for optimisations, sophisticated instrumentation frameworks, such as Valgrind and Pin, have been developed. Both provide only limited facilities for analysing the collected data. This work presents a Valgrind's extension for examining a software applications' dynamic memory profile through live interactive analysis with SQL. The Pico COllections Query Library (pico ql) module maps Valgrind's data structures that contain the instrumented application's memory operations metadata to a relational interface. Queries are type-safe and the module imposes only a trivial overhead when idle. The authors evaluate the proposed approach on ten applications and through a qualitative study. They find 900 kb of undefined bytes in bzip2 that account for 12% of its total memory use and a performance-critical code execution path in the Unix commands sort and uniq. The referenced functions are part of glibc and have been independently modified to boost the library's performance. The qualitative study has users rate the usefulness, usability, effort, correctness, and expressiveness of PICO QL queries compared to Python scripts. The findings indicate that querying with PICO QL incurs lower user effort. [ABSTRACT FROM AUTHOR]

The Unix shell dgsh provides an expressive way to construct sophisticated and efficient non-linear pipelines. Such pipelines can use standard Unix tools, as well as third-party and custom-built components. Dgsh allows the specification of pipelines that perform non-uniform non-linear processing. These form a directed acyclic process graph, which is typically executed by multiple processor cores, thus increasing the processing task’s throughput. A number of existing Unix tools have been adapted to take advantage of the new shell’s multiple pipe input/output capabilities. The shell supports visualization of the process graphs, which can also aid debugging. Dgsh was evaluated through a number of common data processing and domain-specific examples, and was found to offer an expressive way to specify processing topologies, while also generally increasing processing throughput. [ABSTRACT FROM PUBLISHER]

Query formalisms and facilities have received significant attention in the past decades resulting in the development of query languages with varying characteristics; many of them resemble sql. Query facilities typically ship as part of database management systems or, sometimes, bundled with programming languages. For applications written in imperative programming languages, database management systems impose an expensive model transformation. In-memory data structures can represent sophisticated relationships in a manner that is efficient in terms of storage and processing overhead, but most general purpose programming languages lack an interpreter and/or an expressive query language for manipulating interactive queries. Issuing interactive ad-hoc queries on program data structures is tough. This work presents a method and an implementation for representing an application's arbitrary imperative programming data model as a queryable relational one. The Pico COllections Query Library ( pi co ql) uses a domain specific language to define a relational representation of application data structures and an sql interface implementation. Queries are issued interactively and are type safe. We demonstrate our relational representation for objects and the library's usefulness on three large c++ projects. pi co ql enhances query expressiveness and boosts productivity compared to querying via traditional programming constructs. [ABSTRACT FROM AUTHOR]