Search

The Big Data (BD) boom has increased exponentially in recent years, reaching even the most traditional industries. In construction, the migration towards sustainability and new technologies that produce user and environmentally friendly projects is now a requirement in almost every country. Meanwhile, BD technology has become a possible solution to the challenges that the industry faces nowadays with some authors naming this technology as the future of construction. However, despite this reception, studies that explain in detail the factors that favour the adoption of BD are scarce or non-existent and the adoption itself has proven to be a challenge, especially in industries such as construction that are not technology driven. Understanding the critical factors that influence BD adoption has become the focus of many industries that seek to exploit the benefits offered by this technology. Therefore, the aim of this research is to explore the adoption of BD in the construction industry. First, the awareness of the Dominican Republic's construction industry on the BD concept, its characteristics, and benefits was assessed. The key drivers, strategies, and challenges regarding the adoption of BD in the industry were also investigated. A qualitative method was selected to identify these strategies due to the lack of maturity and the scarcity of sources that address the subject. Semi-structured interviews were selected as the data collection tool, and content and thematic analysis were chosen to acquire an in-depth knowledge of the interviews. Endsley's model of situational awareness was adapted to provide a better understanding of the industry's awareness of BD. The sampling technique adopted was non-probabilistic due to some of the specific criteria identified during the secondary data collection process. In the data collection process, 21 interviews were conducted with representatives of 19 organisations with an undoubted presence in the construction market of the Dominican Republic. The results showed that there is an overall basic level of awareness about BD in the construction industry of the Dominican Republic. Moreover, nine key drivers for BD adoption were identified and grouped into internal and external drivers. Additionally, four main strategies or central policies for adopting the technology and seven main challenges were identified. These findings were used to develop an organisation readiness assessment tool and a strategic framework for BD adoption in the construction industry. This study concluded that new technologies such as Big Data (BD) require a change in the industry's culture and the adoption of digital approaches to be fully implemented. The findings of this research provide valuable insights that can help the construction industry adopt BD technology, thus accessing the short and long-term benefits that this technology offers.

This thesis attempts to take volatility research to a new area by exploiting advances in machine learning (ML), natural language processing (NLP), and Big Data analytics. The data underlying the three essays includes high-frequency limit order book (LOB) and daily business news archive for the 23 most liquid NASDAQ stocks over the sample period from 27 July 2007 to 27 January 2022. From 5-minute prices in LOB, I construct the daily realised volatility (RV) of these stocks, which becomes the target for RV forecasting in subsequent empirical studies. In the first essay, I compare the forecasting power of the HAR-family of models, the state-of-the-art go-to group of models for RV modelling and forecasting. I need the best-performing econometric model to serve as the benchmark to compare with ML and NLP models. In the process, this study also becomes the first to test the forecasting power and information content of major LOB variables and news sentiments compiled using the LM dictionary from Loughran and McDonald (2011). After a comprehensive analysis of different model specifications, I conclude that CHAR (continuous HAR) is the best-performing HAR model and that the forecasting patterns are distinctively different between normal volatility days and high volatility days. On high volatility days, a simple news count outperformed all LM news sentiment measures for forecasting RV, and LOB depth outperformed the LOB slope. On normal volatility days, the ask-side of the LOB is more price informative, suggesting that buy-side trades dominate under normal conditions. The second essay focuses on the forecasting power of ML, more specifically, long short-term memory (LSTM) as a pioneer of sequence modelling. 132 LOB variables, nine news variables and 6 HAR variables, together with a variety of model specifications, are tested in this essay. Results show ML outperforms all HAR models (including CHAR) on 90% of the out-of-sample period. Also, SHAP, an explainable AI technique, suggests mid prices, average bid and average ask at all LOB levels improved RV forecasting performance over time. The third essay develops a financial word embedding (FinText) from 15 years of business news archives and compares it to general word embeddings for RV forecasting using stock-related news and general hot news in a simple yet flexible NLP structure. Results show FinText produces more accurate RV forecasts than general word embeddings and significantly improved RV forecasts on high volatility days. This forecasting improvement carries over to normal volatility days when general hot news is used. Again, using SHAP, I was able to compile a list of key phrases from stock-related news and general hot news that moved stock volatility.

BACKGROUND: Scotland has an ageing population, so an increasing responsibility for old age social care. Research elsewhere has demonstrated that several socio-demographic, geographical and health factors predict entry to a care home. Scotland has some key originalities, even compared to other UK countries. Thus, it was important to investigate the factors associated with using social care here. OBJECTIVES: This thesis used routinely collected administrative data to identify socio-demographic, self-rated health, geographical and household factors which were associated with institutional social care use in later life in Scotland. It aimed to investigate whether factors known to be associated with care use for older people through research elsewhere, were generalisable to Scotland. Additionally, it aimed to explore some factors which may not have been investigated elsewhere. METHODS: Linked administrative data from the Scottish Longitudinal Study (SLS), a 5.3% representative sample of the Scottish population (https://sls.lscs.ac.uk/) - provided a longitudinal dataset including people aged 65 years and older, who were still living at home. Information about socio-demographics, household members, informal care, geography and health were gathered from their census responses. Then care status - living in a nursing/care home or not - was determined from the next census. Logistic regression was used to model the associations of these factors with care outcome at follow-up. KEY FINDINGS: The analyses considered two time periods: 1991-2001 and 2001-2011. In the 1991-2001 sample, three lesser researched factors were found to be associated with care use - 1) living in a flat (1.21 (1.04, 1.41))*, 2) recent employment (0.66 (0.53, 0.80)), and 3) population density (Low density 9.71 (7.85, 12.03)); plus an interaction between population density and urban rural classification - with highest attributable risk in low-density city areas (12.83 (9.64, 17.07)). Additionally, patterns for living with different relatives originally found in a Northern Irish study were replicated, with living alone or with siblings associated with increased odds of being in care at follow-up. In the 2001-2011 sample, most associations were consistent, however, this was not the case for the area-based measures such as deprivation and measures of geography, which were no longer associated with care outcome. It was also possible to look at the association of receiving and providing informal care in this later sample, both of which were associated with care outcome (1.81(1.42, 2.29) and 0.70 (0.53, 0.91) respectively). Sex/gender differences were also evidenced for several factors. CONCLUSIONS: Administrative data were used to identify both lesser researched and known factors associated with institutional care use in Scotland, over two different time periods. The factors associated with being in a care home at follow-up in Scotland were broadly similar those evidenced by previous literature in other global-northern populations. The possible explanations for the differences between the samples, which evidence reduced spatial inequality in the later cohort, are discussed, such as the introduction of a new social care policy in 2002 and broader changes to the health and social care landscape in Scotland. Recommendations for future policy and practice are also considered. *(Odds Ratio (95% Confidence Interval)) Note: for the odds ratio the null value is 1. Source: SLS. The help provided by the staff of the Longitudinal Studies Centre - Scotland (LSCS) is acknowledged. The LSCS is supported by the ESRC/JISC, the Scottish Funding Council, the Chief Scientist's Office, and the Scottish Government. The authors alone are responsible for the interpretation of the data. Census output is Crown copyright and is reproduced with the permission of the Controller of HMSO and the King's Printer for Scotland.

Background: Within the given environment of scarce health care resources and increasing demands on the health care system, it is imperative to organise health care in the most efficient manner. Sustainability of the overall system, equity issues concerning access to care and a move to patient-centred care - informing the decision making process by patient needs, are current hot topics when considering care, from curative through to the end of life. The decision making process within UK's health care sector is relying on a standardised approach including the benefits and the costs of interventions and "guarantees" that resources are used in a way, which provide best value. This approach falls short when looking at interventions at the end of life, jeopardising appropriate focus and attention for end of life and palliative care. Aim: The aim of the thesis is threefold: 1. Understanding the current landscape of 'resource use data collection' and deriving costs at the end of life in an UK setting. 2. Reflect on the challenges, benefits and limitations of using administrative data versus trial data in an end of life care setting. 3. Present potential solutions to challenges arising in collecting and analysing cost of end of life care and recommendations for further research. Methods: This work starts by highlighting why end of life care is an outlier when considering the measurement and allocation of health care resources. An update of a systematic review is then presented to identify current methodology used to capture resource use and cost assignment at the end of life. The benefits and limitations of using administrative data or clinical trial data to capture these components is then explored via a series of case studies. The administrative data studies are retrospective whole population secondary care administrative data linkage studies, capturing resources used in the last year prior to death. The first clinical trial is a two-arm parallel group cluster randomised (1:1) trial on pain assessment, the second, a feasibility trial of an Exercise and Nutrition-based Rehabilitation program at the end of life. Results: Most of the results from the included studies are in line with the results of the systematic review. There is a wide variety of perspectives, approaches to resource use collection, costing sources and derivation of costs making a comparison between studies challenging. As the administrative data studies included whole population data, they provided novel insight into the main drivers of secondary care resource use in patients last year of life namely the cause of death being, despite age being a confounding factor, of greater importance than age itself. Further, some equity issues, related to rurality were proven, which was shown to be of bigger influence than deprivation. Different illnesses require various levels of care intensity and are therefore connected to specific care pathways, with cancer care showing an exceptionally high need for secondary care involvement. Both of the studies showed a massive increase in secondary care use with proximity to death, especially within the last three month. The two clinical trials included, showcase different approaches of increasing patients quality of life whilst keeping the impact on resource use low. Both of the trials confirmed secondary care use as the main driver of costs. The second, despite being a small scale feasibility trial, showing a considerable share of resource use falling outside secondary care, hence demonstrate the need to include multiple data sources when aiming to understand the "true cost" of end of life care. Data collection and costing within administrative data studies and clinical trials comes with specific benefits and limitations, some of which could potentially be addressed when combining data generated through trials with administrative data. Conclusion: Fair evidence based decision-making requires comprehensive knowledge of the current state of the system, being conscious of the costs and the benefits of interventions and having a system in place which enables comparing different interventions, which in itself requires a standardised way to capture costs and benefits. This thesis presents various examples of different approaches to resource use collection and applying costs. In order to be able to recommend at least a Scotland wide costing strategy for costing administrative datasets, more research is needed to understand the impact of different costing methods applied to the same dataset. Within clinical trials, collection of resource use data should be kept to a minimum in order to keep patient burden low. The development of a standardized questionnaire is recommended, highlighting the need for adaptability to different trials. Linkage between data collected from individual trials and administrative data is an appealing concept, offering extensive data whilst keeping the patient burden at a minimum. Looking at the current landscape of data protection regulations, there are yet some hurdles to overcome.

Stroke associated pneumonia (SAP) occurs in 8%-14% of people admitted with a stroke. Prevalence of SAP varies considerably across the world, with various frequencies reported among registries and observational studies; however, there is no evidence explaining how SAP varies across stroke units nor what the potential reasons for this variation are. There is also no evidence suggesting how SAP variation affects clinical outcomes. This thesis aims to fill these gaps in knowledge by focusing on 4 different aspects: - to describe the variation of SAP across England and Wales and determine how much of the variation can be accounted by clinical characteristics; - to describe the relationship between SAP and specific stroke care processes in the pre-hospital, hyperacute and acute phases; - to explore the different characteristics used by clinicians to initiate antibiotics in SAP; - to explore the relationship between SAP and clinical outcomes. The first study of this thesis was a registry based cohort study that analysed SAP variation across stroke units in England and Wales, using individual patient level registry data. Median SAP prevalence was 8.5% (IQR 6.1-11.5%). SAP probability was calculated for each stroke unit, as well as variance. The mean and variance of predicted SAP probability decreased from 0.08 (0.68) to 0.05 (0.63). This suggests that patient characteristics account for 5% of the observed variation of SAP across stroke units in England and Wales. The second study of the thesis was also a registry based cohort study that analysed the relationship between timings of certain stroke care processes and the development of SAP across stroke units in England and Wales. Increased times to arrival at a stroke unit, increased time to assessment by a specialist doctor and increased time to assessment by a physiotherapist were associated with increased odds of SAP. Shorter times to thrombolysis were associated with lower odds of SAP. The third study of the thesis was a cross-sectional survey of UK stroke clinicians exploring the different characteristics and thresholds clinicians use to diagnose and initiate antibiotics for suspected SAP. It was an online survey conducted from December 2019 to February 2020. I found that clinicians use known factors such as inflammatory criteria for diagnosis and antibiotic initiation. I also found that the thresholds for each of these factors varied between clinicians, suggesting a lack of standardisation of SAP diagnosis and antibiotic initiation. The final study of the thesis was a registry based cohort study analysing the relationship between SAP and clinical outcomes. I focused on patients whose length of stay in hospital was > 7 days. By focusing on this specific population, potential confounding due to stroke severity on clinical outcomes was mitigated. SAP was consistently associated with increased risk of longer length of stay in hospital with an incidence rate ratio of 1.27 (95% CI 1.25 to 1.30), worse functional outcome at discharge with an odds ratio of 2.9 (2.9 to 3.0) and increased risk of in-hospital mortality with a hazard ratio of 1.78 (1.74 to 1.82). The work presented in this thesis has demonstrated how SAP has important variation across stroke units in England and Wales, and that there are modifiable factors that could be contributing to this observed variation. It has also confirmed the adverse association between SAP and clinical outcomes in real-world stroke unit care. This thesis is the platform for future work to develop a new intervention or tool to reduce the burden of SAP in the UK.

Healthcare is increasingly digitised and recorded, producing huge volumes of 'routinely- collected' data. This data is increasingly viewed as a valuable resource for the improvement of healthcare, although a truly unifying framework to translate data into knowledge and thence change is lacking. From an initial introduction to key concepts and a critique of competing methodologies this thesis identifies a 'systems-approach' as a potential framework to guide data analysis, design, implementation, and overarching project management. The thesis provides evidence for the potential of a systems-approach through two main avenues, a systematic review of the medical improvement literature and the reporting and reflection on two data-intensive improvement initiatives pertaining to the perioperative care of older surgical patients. Four key stages of these case studies are used to determine the role of a systems-approach in the use of routinely-collected healthcare data, with the selection of these phases of work informed by a detailed review of the improvement and epidemiological literature. Evidence for the benefits of a systems-approach is advanced from reflections on project set-up, database construction, development of a causal-model, the design of covariates to capture complex system events (such as integrated working), and the translation of statistical results towards real-world solutions. These results provide evidence that a systems-approach appears to benefit work conducted at a 'project' and 'task' level, can increase transparency in key phases of a data-science workflow, aid in the identification of risks that could lead to biased results, and that an under- standing of the data-generating system appears crucial to understand causal relationships that may be captured in statistical results. These conclusions, as well as the results of each key project phase, form the clear original contributions of this work. The thesis concludes by positing a model of data-intensive improvement as incorporating iterative phases of systems-supported data-science, data- informed design, and data-evidenced implementation which is ready for evaluation in future studies.

The creation of new knowledge from manipulating and analysing existing knowledge is one of the primary objectives of any cognitive system. Out of the Big Data governing Vs, namely Volume, Velocity, Variety, Veracity, Validity, Volatility and Value, the first three are considered the primary ones. Most of the effort on Big Data research has been focussed upon Volume and Velocity, while Variety, "the ugly duckling" of Big Data, is often neglected and difficult to solve. A principal challenge with Variety is being able to understand and comprehend the data in gaining insight. Organisations have been investing in analytics relying on internal and external data to gain a competitive advantage. However, the legal and regulatory acts imposed nationally and internationally have become a challenge. The approach focuses on the use of self-learning systems that will enable automatic compliance of data against regulatory requirements along with the capability of generating valuable and readily usable metadata towards data classification. While for data confidentiality, a framework that utilises algorithmic classification and workflow capabilities is proposed. Such a rule-based system, implementing the corporate data classification policy, will minimise the risk of exposure by facilitating users to identify the approved guidelines and enforce them quickly. Two experiments towards confidential data identification and data characterisation were conducted in evaluating the feasibility of the approach. The focus of the experiments was to confirm that repetitive manual tasks can be automated, thus reducing the focus of a Data Scientist on data identification and thereby providing more focus towards the extraction and analysis of the data itself. In addition to that, a survey with subject matter experts, a diverse audience of academics and senior business executives in the fields of security and data management, was conducted featuring and evaluating a working prototype. The proof-of-concept showcased the model's capabilities and provided a hands-on experience for expert to better understand the proposal. The experimental work confirmed that: a) the use of algorithmic techniques attributed to the substantial decrease in false positives regarding the identification of confidential information; b) evidence that the use of a fraction of a data set, along with statistical analysis and supervised learning is sufficient in identifying the structure of information within it; c) the model for corporate confidentiality is viable and the proposed features of the system are of value. With this proposal, the issues of understanding the nature of data can be mitigated, enabling a greater focus on meaningful interpretation of the heterogeneous data, while at the same time the organisations can secure their data and confirm data confidentiality and compliance.

The existing data protection laws are undergoing a step-by-step process of reviewing, considering, and scrutinising by the law makers and state governments across the regions where existing legislation is believed inadequate. Therefore, the aim should be to reach a consensus amongst like-minded nations to find fundamental discrepancies and variables, resolve contentious/inhibiting factors, with a view to producing proposals to fill the lapses in the law, and conform to a consolidated conceptually meaningful policy framework for sharing with other nations. The main contribution of this research is to develop a Policy Acceptance Model that is meant to be an influencing tool and a guide to develop a data privacy, and security policy. From the purpose angle, it will be an influencing tool for decision making and developing data protection mechanisms and, could be adapted by the states for developing national and regional data protection mechanisms of their own. Cyberspace is not country-specific, not limited to a region or a continent; it belongs to the global community. Therefore, this model is also meant to help develop global level data protection mechanism. Also, the researcher has added new knowledge to the current literature through her publications. These publications addressed contemporary issues such as Huawei row, WhatsApp privacy issue and privacy and security issues associated with COVID-19.

Big data is not new to us. Many efforts are devoted to efficient and parallel query processing of big data. Nevertheless there are still some missing pieces in the technical stack. Existing systems typically assume that there are sufficient resources for query answering, hosted by either on-premise deployed clusters or data warehouses on the cloud. Furthermore, data are always correct and ready to be queried. However none of the above assumptions always holds in practice. This motivates my PhD study to tackle some technical issues with regard to the volume and the veracity of querying big data. The first contribution is a resource bounded approximation scheme (RBAS). We develop a new approximate query processing framework that provides both deterministic and probabilistic accuracy guarantees for generic queries, with the cost of query evaluation in RBAS bounded by a dynamic resource ratio user issued online. Using real-life and synthetic datasets, we show that using only 2% of resources, RBAS improves the accuracy from the state-of-the-art by up to 5 times, and is 3 orders-of-magnitude more efficient than exact query answering. The second contribution is the framework and techniques for querying distributed shared data in a heterogeneous security setting, under which each pair of data owners decide their own protocol to share data with diverse levels of trust. We define query plans by incorporating toll functions determined by data sharing agreements, and formalize query answering as a bi-criteria optimization problem, to minimize both data sharing toll and parallel query evaluation cost. We give both the complexity analysis as well as a set of approximation algorithms to generate efficient distributed query plans. Experimental studies show that our techniques generates efficient query plan under security heterogeneity, outperforms its competitors by 10.27 times in efficiency, and our proposed optimization techniques speed up parallel query evaluation by 3.26 times. Finally the third contribution is coping with inconsistencies in data. We adopt a class of rules for entity enhancing (REE) that embed machine learning predicates, unify entity resolution and conflict resolution, and are collectively defined across multiple relations. We study two related problems using REE: discrepancy detection and entity enhancing (discrepancy fixing). Although both problems are intractable, we develop parallel scalable algorithms and parallel incremental algorithms with parallel scalability. The experimental studies show that REEs improve the accuracy of discrepancy detection and entity enhancing by up to 40% and 57% from its competitors, and our proposed algorithms scale well on a multi-site cluster.

In the age of Big Data Analytics and Covid-19 Apps, the conventional conception of privacy that focuses excessively on the identification of the individual is inadequate to safeguard an individual's identity and autonomy, when she is targeted on the basis of her interdependent social and algorithmic group affiliations. In order to overcome these limitations, this interdisciplinary research develops a theoretical framework of the group right to privacy (GRP), which is based on privacy as a social value (Pᵥ). The quadrumvirate formulation of GRP is articulated on the dual lines of the individual's right as a member of a group and the right of the group itself. An individual's interest in her social identity and her socially embedded autonomous self is protected through GRP₁. The individual's right against algorithmic grouping, GRP₂, is motivated by an interest in group-related aspects of informational self-determination. Thirdly, I provide a non-reductionist account of instances where some organized groups may be entitled to privacy in their own right as GRP₃. Lastly, I articulate the collective interest in Mutual Privacy, understood as an aggregate participatory shared public good which is protected through GRP₄. In all four GRP, I carve out a limited exception for contact tracing by Covid-19 Apps during the extraordinary circumstances of the pandemic while safeguarding against the creation of a new normal of erosion of privacy and the rise of post-pandemic simveillance. To test its efficacy, this theoretical model is critically analysed against the technological challenges posed by Big Data Analytics and Covid-19 Apps. I further examine international privacy legislations to highlight the way this expansive privacy model can be incorporated in the regulatory landscape. In conclusion, this thesis emphasizes that our privacy is not only interdependent in nature, but also existentially cumulatively interlinked and should be protected through the GRP.

The contemporary journey towards data-driven operations is underway in almost all modern organisations, and manufacturing is not an exception. This work addresses several key challenges by: developing hardware and software infrastructure to reliably and economically acquire, process and present industrial data; proposing novel algorithms for analysing Industrial Big Data time series; verifying the proposed analytical techniques through real-world applications in a case study of a commercial bakery. Industry 4.0 heralds a revolution in the field of manufacturing but many of its promises remain undelivered outside experimental contexts. Uptake of digitalised operations is impeded by: capital expenditure, operating expenses, lack of data, the unclear tangible value of undertaking the digitalisation journey, and the challenges of converting data into actionable insights. The Overlay Digitalisation approach is proposed, which uses a modular and open-source architecture to mitigate the characteristic high costs and vendor lock-in of mainstream Industrial IoT platforms. The proposed Industrial Cyber Elements (ICE) Overlay Digitalisation system (spanning from retrofit wireless sensor node to data-handling web services) is developed and deployed in a large commercial bakery. The system has acquired a dataset of more than 40 million observations over the course of 2 years, and has the potential for wide-spread impact due to the permissive licensing of its components. The novel F-QoS metric (a probabilistic measure of wireless network performance requiring only timestamps of transmissions) is developed and applied to assess LoRaWAN ISM-band sensor nodes, and shows that they are a viable solution for the difficult task of data acquisition in a harsh industrial environment. Two novel time series analysis algorithms, Binary Applied Threshold Segmentation (BATS) and Steady-State Extraction Segmentation (SES) are also formulated. Both algorithms are shown to quickly and accurately extract operational regions from unevenly indexed time series, whilst preserving temporal crispness of state transitions. The Notional Alignment Index Lag Sampling (NAILS) method is developed to generate a pairwise comparison between segmented time series, and its ability to work directly with compressed representations extends the scale of analysis that can be undertaken with a fixed compute resource. A novel graph-based methodology is also proposed for analysing time series that arise from batch flow-line production systems, which includes the automatic discovery of system structure and parameters, and the programmatic instantiation of a Discrete Event Simulation. This automated extraction of system knowledge from real-world unlabelled time series data represents a notable step towards automated semantic analysis of industrial data. The depth of knowledge embedded in simple counts of units travelling down a line is revealed by the development of an unbroken chain from raw observations to a self-structuring parametrised system simulation.

The various digital divides have evolved over time and been measured from their earliest days with 'traditional' data sources. These data sources face challenges of data availability, completeness, accuracy, relevance, timeliness and accessibility; thus creating challenges for existing measures of digital divides and the policies that rely on such measures. The aim of this thesis is to investigate the intersection between digital divides and big data, to understand how big data can be used to measure digital divides. Following a systematic literature review (SLR) on big data and digital divides, this study develops a conceptual framework operationalised through a social data science methodology (SDSM) to conduct research using big data to measure digital divides. The SLR explores how big data impacts the way we conceptualise and measure divides. The results confirm the relevance of the topic and reveal the use of big data for measuring traditional and big data-related divides. Two main ways in which big data are used to measure divides are identified: as a complement or as a sole source. Research gaps which are helpful for defining an initial research agenda on measuring divides using big data are also identified. The conceptual framework and the SDSM are used to conduct two empirical cases. The first case analyses broadband divides using aggregated crowdsourced big data, based on an online survey of connectivity speeds from many millions of users. The results of the analyses confirm the existence of broadband divides between high-, middle- and low-income countries. Big data provides a more complete and accurate picture of broadband divides between countries, alongside other benefits. Big data also offers new insights into divides between fixed and mobile networks as well as between download and upload speeds. Big data is also used to calculate two broadband indices and to measure readiness for broadband. The second case analyses knowledge divides using billions of Wikipedia records generated as a by-product of the consumption of contents. The results expose knowledge disparities between countries based on their income and region; with Wikipedia data being more available, accurate in some ways, timely, and accessible compared to traditional data. In addition, these data allow the identification of detailed consumption volumes, per capita estimates, device preferences, and the analysis of language divides that cannot be measured with traditional sources. Combined with Wikipedia production data, a new index is created, helping provide additional insights into knowledge divides. Overall, the results of this study show that big data is changing the way we conceptualise and measure digital divides. The empirical cases demonstrate that big data can be used as a complement of traditional measures with some advantages when compared to traditional data. Big data offers new insights into digital divides that have not been measured due to the limitations of traditional data. Additional information that can be used to inform policy decisions --such as the estimation of indices-- can be obtained thanks to the additional data points included in these big data sources. Nevertheless, the limitations of big data are also acknowledged thus helping analysts decide on the use of these type of sources when measuring digital divides and for policy making. A Big Data Usage Decision Tool is presented to be used as all or part of ex-ante evaluation tools.

This research examines the relationship between Big Data and competition law. Initial findings reveal that accumulating personal information through data collection and acquisition methods benefits consumers considerably. Free of charge, fast and personalised services and products are offered to consumers online. Collected data is now an indispensable part of online businesses to the point that a new economy, a data-driven sector, has emerged. Many markets such as the social network, search engine, online advertising and e-commerce are regarded as data-driven markets where the utilisation of Big Data is a requisite for the success of operations. However, on the other side of the medallion, the accumulation and utilisation of data bring competition law concerns since they contribute to market power in the online world. Therefore, a few technology giants have gained unprecedented market power due to the Big Data accumulation, indirect network effects and the creation of online ecosystems. The sector players such as Google, Amazon and Facebook have established a strong market presence and dominance in their respective markets. As technology giants have billions of consumers worldwide, data-driven markets are truly global. In these data-driven markets, technology giants abuse their dominant positions. However, the current competition law tools seem ineffective in addressing market power and assessing abusive behaviour related to Big Data. To address this challenge, a novel approach to the data-driven sector is developed through the application of competition law rules. The novel approach proposed by this research justifies that the current and potential conflicts can be mitigated by extending the competition law assessment beyond the current competition law tools. Thus, the need for a modernised and unified approach to the Big Data related competition issues is identified. Ultimately, this research contributes to EU Competition Law by promoting new legal tests in addressing the market power of technology giants and assessing abusive behaviour in data-driven markets. Additionally, the need for cooperation between competition and data protection authorities is also nuanced.

This thesis outlines the design and development of CIRAFm (Cancer Integromics Research Application Framework). CIRAFm is a software framework for interactively analysing biomedical data within cancer research. It provides a no-code browser platform which allows translational researchers to conduct, record, share their analysis both remotely and in real-time. The flexibility and extensibility of the platform as a result of its modular architecture allows CIRAFm to easily support multiple technologies including deep-learning capabilities and various programming languages within its no-code analytical spaces. The traditional and AI-based analytical capabilities of CIRAFm were tested and validated through a set of use cases centring on a published CRC data cohort and a CRC tissue image dataset. Through the testing of multiple use cases CIRAFm has shown the potential to support a wide range of analytics from EDA to AI-based methods. It also addresses several key of interest within the current biomedical research landscape including result reproducibility and secure sharing, democratizing access to biomedical analytics, and providing greater flexibility to researchers and scientists with their analytical workflows.

Increasingly, there is a growing trend for inter-organisational collaborative Big Data sharing and analysis. For efficiency reasons, such Big Data analysis is usually carried out by using distributed computing services deployed in public clouds. Executing Collaborative Big Data Computation (CBDC) in a Multiple Public Cloud (MPC) environment introduces some open issues. One of these issues is how to maximise security protection level with minimum overhead costs. We set to investigate these issues based on the authentication property as authentication is the first line of defence in any computing systems. The investigation has led to the design, prototype, and evaluation of a novel authentication solution that takes into account of the characteristics of the underlying system. To this end, this thesis has made the following contributions. Firstly, the thesis has formulated a generic use case model for CBDC-MPC. This model captures an extreme form of distributed computation where multiple collaborators jointly perform CBDC on shared datasets using an example distributed computing framework, MapReduce (MR), deployed in an MPC environment. The model is used to gain a thorough understanding of the threats in relation to impersonation, unauthorised access, and alteration to data in the context and guide the design of an effective, efficient, and scalable authentication solution for distributed systems. Secondly, the thesis has proposed a novel authentication framework for CBDC-MPC. The framework, called the Multi-domain Decentralised Authentication (MDA) framework, consists of two further novel components, the Multi-factor Interaction based Entity Authentication (MIEA) framework and the Communication Pattern based Data Authentication (CPDA) framework. The MIEA framework provides risk-aware entity authentication to every interaction during the entire execution cycle of a data processing job. The framework has been analysed and evaluated both theoretically and experimentally. The analysis and evaluation results demonstrate that MIEA provides a stronger level of entity authentication but with the same level of overhead cost compared with Kerberos, one of the most used entity authentication protocols in a distributed computing environment. The CPDA framework provides data authenticity and non-repudiation of origin for every data object processed by the underlying system. To maximise the protection level while minimising the overhead cost, a novel idea of communication pattern based aggregations of authentication data (generation and verification operations) and communication is used in conjunction with multiple cryptographic schemes. The theoretical and experimental evaluation results show that the CPDA approach offers the strongest level of data authenticity protection but reduces the overhead cost by up to 67% in comparison with the most related solution that digitally signs every object individually. The results demonstrate that the idea of tailoring the design of an authentication solution in line with the characteristics of the underlying system brings much benefit in terms of supporting efficient and scalable authentication in a large-scale distributed system.

While large-scale distributed data processing platforms have become an attractive tar- get for query processing, these systems are problematic for applications that deal with nested collections. Programmers are forced either to perform non-trivial translations of collection programs or to employ automated flattening procedures, both of which lead to performance problems. These challenges only worsen for nested collections with skewed cardinalities, where both handcrafted rewriting and automated flattening are unable to enforce load balancing across partitions. In this work, the TraNCE compilation framework is proposed that translates a program manipulating nested collections into a set of semantically equivalent shredded queries that can be efficiently evaluated. The framework employs a combination of query compilation techniques, an efficient data representation for nested collections, and automated skew-handling. Biomedical case studies are presented that outline research and clinical applications for the platform, including data integration support for building feature sets for classification. An extensive experimental evaluation is provided using both synthetic and real-world dataset from the biomedical domain. The evaluation shows that the system is capable of outperforming the common alternative, based on "flattening" complex data structures, and runs efficiently when alternative approaches are unable to perform at all.

As the complexity of distributed analytics systems evolves over time, more configuration parameters get exposed for tuning. While these numerous parameters allow users more control over how their workloads are executed, this flexibility comes at a cost, since finding the right configurations for such systems in a cost-effective way becomes challenging. In practice, several factors contribute to the complexity of tuning the configuration of those systems: the large configuration space, the diversity of the served workloads (each workload possibly requiring a different resource allocation strategy to run optimally), and the dynamic characteristics of these systems' environment (e.g., increase in input data size, changes in the allocation of resources). Paradoxically, existing solutions for workload tuning either assume static tuning environment or workloads that are inexpensive to run (i.e. requiring hundreds of execution samples). Recently, Bayesian Optimisation (BO) strategies have been applied as a solution to enable efficient autotuning. They build a probabilistic model incrementally to predict the impact of the parameters on performance using a small number of execution samples. The incrementally constructed BO model is used to guide the tuning process and accelerate convergence to a near-optimal configuration. Unfortunately, for distributed analytics systems, the configuration space is too large to construct a good model using traditional BO, which fails to provide quick convergence in high dimensional configuration space. I argue that cost-effective tuning strategies can only be developed when taking into account: the frequent changes that can happen in the analytics workload/environment, the amortization of tuning costs and how this influences tuning profitability, the high dimensionality of configuration space and the need to cater for diverse workloads. To tackle these challenges, I propose Tuneful, an efficient configuration tuning framework for such expensive to tune systems. It works efficiently both initially (when little data is available) as well as later (as more tuning knowledge is acquired). It starts with learning workload-specific influential parameters incrementally and tunes those only, then when more tuning knowledge becomes available, it detects similarity across workloads and utilizes multitask BO to share the tuning knowledge across similar workloads. I show how augmenting the BO approach with parameters' significance and workload similarity characteristics enables an efficient configuration tuning in high dimensional configuration space. Over diverse analytics workloads, this significantly accelerates both configuration tuning and cost amortization, saving search time by 2.7-3.7X at median compared to the-state-of-the-art approaches.

The growth in the number and complexity of new smart devices has been exponential in recent years. With the increasing understanding and application of artificial intelligence and machine learning, smart devices have been used in creating new opportunities for intelligent solutions that can enable services suited for smart cities, autonomous systems and ubiquitous systems monitoring and control. Smart devices, including mobile devices, usually have a small-scale factor and have limited space for batteries, computing, and memory resources. This places a demand for such devices to strictly manage the use of resources to remain in operation for a longer period. In current and upcoming applications of smart devices, such as in the IoT, a network of devices, commonly referred to as a wireless sensor network, needs to gather data by sensing, computing the data, and reporting the information to a base station. Often these data is huge in size and transmitting all the data to the base station would drain the devices of their limited resources. However, the consumption of resources within the device is directly related to the communication and routing algorithm used across the network by each device. Thus, to improve the network's performance through extending its lifetime and addressing more applications than it was specifically built for, the network needs to be sensitive to changes in the context of the application and be able to dynamically select the appropriate routing algorithm to apply based on various performance objectives. The aim of this research involved the investigation and analysis of the problem, including a study of relevant literature and supporting theory, and culminated in the development of such an adaptive model that can dynamically manage a set of smart mobile devices. It included the investigation of the behaviour of a set of smart devices and their data management approach, while identifying the factors that determined their performance metrics. Metrics considered included energy consumption, bandwidth, and latency. With this knowledge as foundation, an adaptive model with capability to dynamically determine the optimal data management approach in a collection of devices was designed, developed, and evaluated. Various unique single and complex scenarios (scenarios with more than one application running) were used in an evaluation of the model and the results of this process proved that the model outperformed the current state of the art.

Prostate cancer is highly heterogeneous, with the potential for multifocality with phenotypically and genetically distinct tumour foci. Additionally, cellular heterogeneity has the ability to harbour phenotypically diverse cancer cell types. Prostate cancer needs to be explored across different levels to enhance our understanding of disease progression. This can be achieved through investigating progression at a cellular level, before exploring whole patient genomics, and then across larger patient cohorts. Technological advancements within genomics are changing how we understand prostate tumours and their heterogenous nature, which in turn have the potential to better deliver personalized medicine. This thesis questions whether bioinformatics tools can assess intra-tumour heterogeneity via alignment-free methods and identify novel prostate cancer biomarkers associated with advanced disease across these different levels. Exploring isolated prostate cellular sub-populations allows for novel biomarkers and targetable pathways to be identified towards cancer stem-cell (CSC) biology. Towards the patient level, intra-patient heterogeneity can be assessed through next generation sequencing by adopting multi-regional sampling via whole exome sequencing. Expanding to the cohort level, AI learning algorithms in particle swarm optimization (PSO) can explore and visualize, in real time, the most informative and succinct gene signatures from transcriptomic datasets. Differentially expressed genes (DEGs) from the isolated prostate side-populations (SPs) have shown these prostate SPs to be genetically distinct from terminally differentiated cells. DEGs were also used as input gene signatures for identifying FDA-approved drug targets for potential repurposing towards hormone manipulated distal SP (DSP) samples when compared against non- androgen deprivation therapy DSP samples. Using novel alignment-free phylogenetic analysis, we assessed intra-patient heterogeneity across different patients' multi-regional samples and identified cases where divergent branching from phylogenetic trees validated different lesions within a patient's prostate and the impacts this may have on drug targeting. In comparison with standard PSO algorithms, our enhanced binary PSO (EBPSO) Flask mini-framework and Python module produces similar to better accuracy for gene signature discovery using less genes and at a much faster runtime on clinical cancer transcriptomics cohorts. The key genes within these signatures reveal functions that were well correlated to their cancer type. Familial adenomatous polyposis (FAP), a key gene within the candidate signature produced from the FASTMAN prostate cancer dataset, has been associated with prostate cancer metastatic disease and is seen to be expressed in prostate cancer stroma. We have proposed that recurring lesions originate from the DSP cells following hormone manipulation, which in turn will need different therapeutic targets from the primary tumour. Multi-regional sampling in combination with novel alignment-free bioinformatics software such as those assessing intra-patient heterogeneity is a valuable option to help deliver personalised medicine through more robust efforts in the discovery of targetable pathways and therapeutic strategies. Our EBPSO Flask mini-framework can aid medical data research for novel discoveries at an efficient pace with lower computational needs, showing real time visualizations and analysis statistics towards discovery.

The advent of big data about a decade ago, coupled with its processing and storage challenges gave rise to the development of a multitude of data-intensive frameworks. These distributed parallel processing frameworks can be used to process petabytes of data stored in a cluster of computing nodes. Companies and organisations can now process massive amounts of data to drive innovation and gain a competitive advantage. However, these new paradigms have resulted in several research challenges due to their inherent difference from the more mature traditional data processing and storage systems. Firstly, they are comparatively more modern, supporting the execution of a wide variety of new data-intensive workloads with varying performance requirements. Therefore, there is a clear need to study and standardise ways to benchmark and compare them to identify and improve performance bottlenecks. Secondly, they are highly configurable; enabling users the freedom to tune the execution environment based on the application's performance requirements. However, this freedom and the ubiquity of the configuration parameters present an additional challenge by shifting the tuning and optimisation responsibilities of these numerous configuration parameters to the users. To address the above broad challenges, in this research, we enabled a grey-box benchmarking and performance modelling framework focusing on two of the most common communication patterns for data-intensive applications. The use of communication patterns allowed us to classify and study varying but related data-intensive workloads using the same sets of requirements. Furthermore, we enabled a multi-objective performance prediction framework that can be used to answer various performance-related questions such as the time it takes to execute an application, the best configuration parameters to satisfy constraints such as deadline, and recommendation of optimal cloud instances to minimise monetary cost. To gauge the generality of this work, we have validated the results on two internal clusters, and the results are consistent across both setups. We have also provided a REST API and web implementation for validation. The primary take way result is that the research showcase a comprehensive approach that can be used to benchmarking and modelling the performance of data-intensive applications.

My thesis explores fiction's response to the question of how the contemporary internet is reshaping subjectivity. Through case studies on Joshua Cohen's Book of Numbers (2015), Nicola Barker's H(A)PPY (2017), David Foster Wallace's The Pale King (2011), and Neal Stephenson's Fall; or, Dodge in Hell (2019), my thesis identifies the extent to which fiction can encourage contestation of a particular understanding of digital culture. I loosely consider these novels as a subset that has evolved from the "systems novels" defined by Tom LeClair in the 1980s-novels which seek to unpack how human beings are formed, informed, and deformed by the intersection and collision of social systems. To this end, I explore how fiction can reveal the systems which underpin digital culture and, by revealing them to be constructed, encourage the reader to consider how they may be reconstructed. In order to explore how these systems are depicted in fiction, I define the key characteristics of what I have called "algorithmic manipulation"-specific characteristics of algorithms which contribute to digital culture's ability to reshape subjects and subjective experience. This framework draws upon existing digital culture scholarship, including the work of Antoinette Rouvroy, Shoshana Zuboff and Byung-Chul Han, as well as upon the theory of computation. But on a broader level, my framework draws upon and extends Michel Foucault's bio-power concept, to consider how algorithms herd subjects en masse, and Ludwig Wittgenstein's philosophy of language. My use of the latter interrogates to what extent the fact that algorithms are inscribed in code has had an influence on digital culture. I argue that algorithmic manipulation functions through language and through narrative, and it is on these two fields that literature can most effectively engage with and encourage the contestation of digital culture.

The world is producing digital data at a rapid pace and this led to a new way of seeing data as Big Data. Big Data refers to large numbers of large datasets that typically include complex, various, rapidly-changing data of uncertain quality that (usually) need preprocessing before analysis. Data integration over Big Data gives rise to the challenge of correlating unruly and heterogeneous repositories of data sources. In this thesis, our focus is on integration techniques for Big Data, more specifically on generating mappings over large repositories of heterogeneous and autonomous datasets. A schema mapping generation algorithm constructs views for populating a target database schema from source schemas. We have designed, developed, and validated techniques for generating schema mappings over autonomous data sources for which scant information is available, and for complex multi-relation, constrained target schemas, at scale. Our proposed algorithm is called Dynamap and has at its core the dynamic programming paradigm for performing the search over the space of mappings. The mappings are built in a bottom-up fashion, where the merge operators are chosen based on profiling information on the sources, i.e., candidate keys and (partial) inclusion dependencies. We have employed Dynamap in three main types of experiments: (i) with the state-of-the-art integration scenario generator, thus, showing that it can handle scenarios that are expected to be tackled by mapping generation algorithms; (ii) with variations of real-world scenarios that come from different domains with autonomous sources, showing that it can handle integration problems from real datasets; and (iii) with stress-test scenarios showing that our algorithm can handle scenarios where the input comprises hundreds of data sources.

Big Data is changing the traditional way of data collection. As an example of Big Data, mobile phone signal data can be used in geography and urban planning analysis. The approximate spatial position of the mobile-phone user can be calculated according to the signal data sent by a mobile-phone and the location of the base station. This research demonstrates that it is possible to use the mobile-phone signal data to calculate the spatial distribution of urban population. It also tests the suitability of existent metrics and tries to evaluate change by creating new metrics. In Shanghai, by using existent metrics, this study finds that the spatial distribution of the city’s population calculated by mobile-phone signal data generally corresponds to the demographic census. Moreover, this research also employs mobile phone signal data to analyse the spatial correlation between facilities, housing price and other variables and population density distribution in Shanghai. Finally, this study calculates the individual’s origin-destination (OD) flows of Shanghai, gets the job housing distance, and identifies the difference of job-living spatial movement pattern of Shanghai residents. This research also intends to figure out the factors that cause the difference in job-housing spatial relationships between four groups of people in different types of urban functional areas. This research corroborates David Harvey’s time-space compression theory and spatial fix theory by using Big Data set. Going beyond that, it extends these theories by arguing the interchangeability between exploitation in the time dimension, and that in the spatial dimension, resulting in dual exploitation. This study has important policy implications for optimizing the urban spatial system of Shanghai, as it advocates that in addition to strengthening the enforcement of labour law, the government also needs to improve the public service such as strengthening the public transportation systems capacity, and construct affordable houses, so as to alleviate the employees’ sufferings caused by temporal and spatial exploitation.

Poverty and disadvantage in the United States are commonly defined in terms of low income. Via this measure, in 2015, over 40 million people were deemed as poor. This definition and subsequent measurement neglect the multidimensional nature of the phenomena. It has been acknowledged that this reductionist measure is insufficient to capture many dimensions of hardship beyond the economic. However, there have been few attempts at quantifying multidimensional disadvantage in the United States. The aim of this thesis is to quantify multidimensional disadvantage by applying the concept of social exclusion to ‘big’ American data, the United States Census Bureau-produced American Community Survey (ACS) Public Use Microdata Sample (PUMS) file for 2015 that contains over 2.3 million sample members. Social exclusion, as a concept, theoretically addresses many of the limitations of the official measure. In particular, it offers a multidimensional conceptualisation of disadvantage. This concept, however, is substantially under-researched in the United States. In order to apply the concept to a context in which it is rarely used, social exclusion is measured and defined based on the theoretically derived framework, the Bristol Social Exclusion Matrix (B-SEM). This framework identifies three interconnected domains of social exclusion: resources, participation and quality of life. The substantive and methodological objectives of this thesis are threefold: 1) to empirically derive the factors of disadvantage in the United States by applying B-SEM to indicators found within the ACS PUMS, 2) to assess the relationship between sociodemographic variables and the dimension(s) of disadvantage, and 3) to explore state-level variation in disadvantage across the United States. An exploratory factor analysis was used to empirically derive the factors of disadvantage in the United States. The results produced three distinct factors: ‘labour force participation,’ ‘economic security,’ and ‘marriage as a social resource.’ This highlights that disadvantage in the United States is indeed multidimensional, with income representing one component of one factor. Therefore, a focus on a lack of income is incomplete to fully understand disadvantage in the United States. Six ordinary least squares (OLS) multivariate regression models were used to analyse the relationships between the sociodemographic characteristics, age, race, gender, and citizenship status and intersectional characteristics (the intersection between gender and race). In the non-intersectional models, it was found that these characteristics explain more variation in the ‘labour force participation’ model, compared to the other two dimensions. In the intersectional model, however, over three times the variation is explained in the ‘economic security’ model, compared to the other two dimensions. The results highlight that different individuals do experience disadvantage differently, showcasing the importance of recognising and addressing multiple forms of disadvantage. Twelve multilevel models were used to assess if there was variation in the dimensions of disadvantage across the United States, if that variation held controlling for sociodemographic characteristics, and if the relationships between the individual characteristics and the dimensions of disadvantage varied across states. The models demonstrated that there was state level variation in each dimension of disadvantage across the United States and that variation persisted once individual characteristics were controlled for. In addition, it was found that the effect of gender varies significantly across states for each dimension of disadvantage. These results highlight the importance of context in understanding disadvantage and shed light on an important role the state plays in reducing and preventing disadvantage. These results have important implications for policies designed to alleviate disadvantage in the United States. In addition to expanding all income-based benefits at least to individuals who are 250% above the federal poverty line, state governments should promote the provision of health care to all members of their respective populations and provide incentives that encourage educational attainment.

A huge amount of data is generated daily by digital technologies such as social media, web logs, traffic sensors, on-line transactions, tracking data, videos, and so on. This has led to the archiving and storage of larger and larger datasets, many of which are multi-modal, or contain different types of data which contribute to the problem that is now known as "Big Data". In the area of Big Data, volume, variety and velocity problems remain difficult to solve. The work presented in this thesis focuses on the variety aspect of Big Data. For example, data can come in various and mixed formats for the same feature(attribute) or different features and can be identified mainly by one of the following data types: real-valued, crisp and linguistic values. The increasing variety and ambiguity of such data are particularly challenging to process and to build accurate machine learning models. Therefore, data heterogeneity requires new methods of analysis and modelling techniques to enable useful information extraction and the modelling of achievable tasks. In this thesis, new approaches are proposed for handling heterogeneous Big Data. these include two techniques for filtering heterogeneous data objects are proposed. The two techniques called Two-Dimensional Similarity Space(2DSS) for data described by numeric and categorical features, and Three-Dimensional Similarity Space(3DSS) for real-valued, crisp and linguistic data are proposed for filtering such data. Both filtering techniques are used in this research to reduce the noise from the initial dataset and make the dataset more homogeneous. Furthermore, a new similarity measure based on intuitionistic fuzzy set theory is proposed. The proposed measure is used to handle the heterogeneity and ambiguity within crisp and linguistic data. In addition, new combine similarity models are proposed which allow for a comparison between the heterogeneous data objects represented by a combination of crisp and linguistic values. Diverse examples are used to illustrate and discuss the efficiency of the proposed similarity models. The thesis also presents modification of the k-Nearest Neighbour classifier, called k-Nearest Neighbour Weighted Average (k-NNWA), to classify the heterogeneous dataset described by real-valued, crisp and linguistic data. Finally, the thesis also introduces a novel classification model, called FCCM (Filter Combined Classification Model), for heterogeneous data classification. The proposed model combines the advantages of the 3DSS and k-NNWA classifier and outperforms the latter algorithm. All the proposed models and techniques have been applied to weather datasets and evaluated using accuracy, Fscore and ROC area measures. The experiments revealed that the proposed filtering techniques are an efficient approach for removing noise from heterogeneous data and improving the performance of classification models. Moreover, the experiments showed that the proposed similarity measure for intuitionistic fuzzy data is capable of handling the fuzziness of heterogeneous data and the intuitionistic fuzzy set theory offers some promise in solving some Big Data problems by handling the uncertainties, and the heterogeneity of the data.

Social psychology has a long tradition of studying the personality traits associated with subjective well-being (SWB). However, research often depends on a priori but unempirical assumptions about how to (a) measure the constructs, and (b) mitigate confounded associations. These assumptions have caused profligate and often contradictory findings. To remedy, I demonstrate how a computational psychology paradigm-predicated on large online data and iterative analyses-might help isolate more robust personality trait associations. At the outset, I focussed on univariate measurement. In the first set of studies, I evaluated the extent researchers could measure psychological characteristics at scale from online behaviour. Specifically, I used a combination of simulated and real-world data to determine whether predicted constructs like big five personality were accurate for specific individuals. I found that it was usually more effective to simply assume everyone was average for the characteristic, and that imprecision was not remedied by collapsing predicted scores into buckets (e.g. low, medium, high). Overall, I concluded that predictions were unlikely to yield precise individual-level insights, but could still be used to examine normative group-based tendencies. In the second set of studies, I evaluated the construct validity of a novel SWB scale. Specifically, I repurposed the balanced measure of psychological needs (BMPN), which was originally designed to capture the substrates of intrinsic motivation. I found that the BMPN robustly captured (a) dissociable experiences of suffering and flourishing, (b) more transitive SWB than the existing criterion measure, and (c) unique variation in real-world outcomes. Thus, I used it as my primary outcome. Then, I focussed on bivariate associations. The third set of studies extracted pairs of participants with similar patterns of covarying personality traits-and differing target traits-to isolate less-confounded SWB correlations. I found my extraction method-an adapted version of propensity score matching-outperformed even advanced machine learning alternatives. The final set of studies isolated the subset of facets that had the most robust associations with SWB. It combined real-world surveys with a total of eight billion simulated participants to find the traits most prevalent in extreme suffering and flourishing. For validation purposes, I first found that depression and cheerfulness-the trait components of SWB-were highly implicated in both suffering and flourishing. Then, I found that self-discipline was the only other trait implicated in both forms of SWB. However, there were also domain-specific effects: anxiety, vulnerability and cooperation were implicated in just suffering; and, assertiveness, altruism and self-efficacy were implicated in just flourishing. These seven traits were most likely to be the definitive, stable, drivers of SWB because their effects were totally consistent across the full range of intrapersonal contexts.

This thesis is focused on the development of computationally efficient procedures for regression modelling with datasets containing a large number of observations. Standard algorithms be prohibitively computationally demanding on large n datasets, and we propose and analyse new computational methods for model fitting and selection. We explore three different generic strategies for tall datasets. Divide and conquer approaches split the full dataset into subsets, with the subsets then being analysed independently in parallel. The subset results are then pooled into an overall consensus. Subsampling based methods repeatedly use minibatches of data to estimate quantities of interest. The third strategy is 'sketching', a probabilistic data compression technique developed in the computer science community. Sketching uses random projection to compress the original large dataset, producing a smaller surrogate dataset that is less computationally demanding to work with. The sketched dataset can be used for approximate inference. We test our regression algorithms on a number of large n genetic datasets, aiming to find associations between genetic variants and red blood cell traits. Bayesian divide and conquer and subsampling methods have been studied in the fixed model setting but little attention has been given to model selection. An important task in Bayesian model selection is computation of the integrated likelihood. We propose divide and conquer and subsampling algorithms for estimating the integrated likelihood. The divide and conquer approach is based on data augmentation, which is particularly useful for logistic regression. The subsampling approach involves constructing upper and lower bounds on the integrated likelihood. Lower bounds can be formed using variational Bayes techniques and we show how subsampling can be used to estimate an upper bound on the integrated likelihood. Sketching algorithms generate a compressed set of responses and predictors than can then be used to estimate regression coefficients. Sketching algorithms use random projections to compress the original dataset and this stochastic generation process makes them amenable to statistical analysis. We examine the statistical properties of sketching algorithms, which allows us to quantify the error in the coefficients estimated using the sketched dataset. The proportion of variance explained by the model proves to be an important quantity in choosing between alternative sketching algorithms. This is particularly relevant to genetic studies, where the signal to noise ratio can be low. We also investigate sketching as a tool for posterior approximation. The sketched dataset can be used to generate an approximate posterior distribution over models. As expected, the quality of the posterior approximation increases with the number of observations in the sketched dataset. The trade-off is that computational cost of sketching increases with the size of the desired sketched dataset. The main conclusion is that impractically large sketch sizes are needed to obtain a tolerable approximation of the posterior distribution over models. We test the performance of sketching for posterior approximation on a large genetic dataset. A key finding is that false positives are a major issue when performing model selection. Practical regression analysis with large n datasets can require specialised algorithms. Parallel processing, subsampling and random projection are all useful tools for computationally efficient regression modelling.

The rare disease pulmonary arterial hypertension (PAH) results in high blood pressure in the lung caused by narrowing of lung arteries. Genes causative in PAH were discovered through family studies and very often harbour rare variants. However, the genetic cause in heritable (31%) and idiopathic (79%) PAH cases is not yet known but are speculated to be caused by rare variants. Advances in high-throughput sequencing (HTS) technologies made it possible to detect variants in 98% of the human genome. A drop in sequencing costs made it feasible to sequence 10,000 individuals including 1,250 subjects diagnosed with PAH and relatives as part of the NIHR Bioresource - Rare (BR-RD) disease study. This large cohort allows the genome-wide identification of rare variants to discover novel causative genes associated with PAH in a case-control study to advance our understanding of the underlying aetiology. In the first part of my thesis, I establish a phenotype capture system that allows research nurses to record clinical measurements and other patient related information of PAH patients recruited to the NIHR BR-RD study. The implemented extensions provide a programmatic data transfer and an automated data release pipeline for analysis ready data. The second part is dedicated to the discovery of novel disease genes in PAH. I focus on one well characterised PAH disease gene to establish variant filter strategies to enrich for rare disease causing variants. I apply these filter strategies to all known PAH disease genes and describe the phenotypic differences based on clinically relevant values. Genome-wide results from different filter strategies are tested for association with PAH. I describe the findings of the rare variant association tests and provide a detailed interrogation of two novel disease genes. The last part describes the data characteristics of variant information, available non SQL (NoSQL) implementations and evaluates the suitability and scalability of distributed compute frameworks to store and analyse population scale variation data. Based on the evaluation, I implement a variant analysis platform that incrementally merges samples, annotates variants and enables the analysis of 10,000 individuals in minutes. An incremental design for variant merging and annotation has not been described before. Using the framework, I develop a quality score to reduce technical variation and other biases. The result from the rare variant association test is compared with traditional methods.

Big Four and mid-tier audit firms have made huge investments in developing and acquiring tools that are used to exploit Big Data (BD) environments. Further, audit regulatory agencies are also interested in understanding how audits in such environments are being conducted and the effects of these tools. Therefore, this thesis situates the deployment of tools to extract BD, referred to as Big Data Analytics (BDA) and their significance for the conduct of external audits. . It addresses two important areas, namely the promotion of BDA in the audit field and its embedding in the audit process. The promotion of BDA in the audit field is understood through the construct of rhetorical strategies, whereas the embedding of BDA in the audit process is explored using the constructs of identity regulation and affordance. These constructs are drawn from new institutional theory and the sociomateriality perspective with the objective of understanding how BDA, as audit technology, is implicated in the social relevance of auditing. The evidence is collected from 27 semi-structured interviews with individuals dealing with BDA in the audit field and beyond, from publicly available textual data from audit firms and from observations of audit firms' proprietary BDA tools. The findings indicate that, in promoting BDA, audit firms are using various discursive strategies either to stimulate more public trust in auditors' commitment to audit quality or to add value to their clients. In this regard, audit firms use teleological, ontological, historical and value-based arguments to show that BDA addresses concerns associated with audit quality. On the other hand, audit firms use cosmological arguments and arguments based on value added to show that BDA could enhance the operational efficiencies of their clients. However, the firms face challenges in relation to issues of data security and lack of understanding of the meaning of BD and the context in which BDA should be used in the audit. Further, to embed BDA, audit firms use discursive strategies aimed at 'winning hearts and minds' of auditors at all levels of the hierarchy within firms. This involves developing the identity of auditors as both data analysts and business advisors to the client who generate value for their firms by exploiting opportunities associated with BDA. Therefore, auditors are encouraged to show commitment to using BDA during audits through recruitment, capacity building, incentives, personal reviews and the industrialisation of BDA. Finally, BDA is implicated in the reconfiguration of data processes. Auditors argue that BDA offers affordances which enable them to achieve greater coverage in terms of operational scope and depth. Such affordances, in particular the use of visualisation tools, could have implications for their professional judgements because audit evidence can be manipulated and presented in various dimensions to clients. However, BDA constrains auditors who may not have the relevant expertise to operate some of its functionalities. The study contributes by providing early empirical accounts of the latest and most significant developments in audit technology. Further, it adds to prior literature on changes in audit technology by highlighting that such technological transformation also requires that auditors reconsider their professional identities. This leads to a revisited focus on not only what auditors do but also who they are (need to be). Finally, in studying technological changes in audit, the research illustrates the importance not only on how auditors make use of the technology but also the properties of the technology itself (its socio-material characteristics) as a significant influence on how it is embedded and used in practice.

Consumers' interactions with social media have become inescapable. The adoption and adaptation of these platforms are rapidly increasing. It is not known as to what underlying factors drive consumers' social media adoption, subsequent interactions with social media platforms and how they respond to the consequences of their digital footprints. Little attention is given to consumers' joint attitudinal components (cognitive and affective) as antecedents of a composite process of technology adoption and adaptation. Thus, this research aims to examine consumers' social media adoption, adaptation and behavioural outcomes as a composite process based on their joint attitudinal components. An online survey strategy under a positivist research paradigm with a deductive research approach was undertaken. Under an explanatory quantitative research method, data was collected through a structured questionnaire, administered to a random sample of social media consumers. Data was analysed using Structure Equation Modelling. The findings confirm that consumers' joint attitudinal components are the antecedents of social media adoption, adaptation and behavioural outcome as a composite process that is influenced by cognitive (Perceived Opportunity, Perceived Social Influence and Perceived Control) and affective (Enjoyment, Self-Enhancement, Trust and Fear) attitudinal components. Consumers engage in adaptation behaviour of Exploration to Maximise or Exploitation to Satisfice Social Media Benefits if they have a favourable attitude towards the consequences of their digital footprints and Exploration to Revert from social media or Avoidance of Social Media adaptation behaviour if their attitude is unfavourable. Each adaptation behaviour leads to a specific outcome such as carelessness and carefreeness when consumers engage in exploration to maximise or exploitation to satisfice social media benefits and cautiousness and consciousness when they engage in social media avoidance and exploration to revert adaptation behaviour respectively. This research makes the following contributions to literature. It develops a nomological model of technology adoption, adaptation and behavioural outcome as a composite process. It offers the joint attitudinal components (cognitive and affective) as antecedents to the composite process. It introduces consumers' adaptation behavioural efforts to cope with the consequences of their digital footprints and analyses the impact of technology adoption and adaptation on behavioural outcomes. The research helps marketers in segmenting, profiling and targeting customers. It allows marketers and information systems managers to focus on consumers' joint attitudinal components by identifying and analysing the key antecedents of the technology adaptation which can provide useful insights for future product/service development. In addition, it enables marketers to analyse consumers' attitudinal response to their privacy and security of their digital footprints, and offers useful policy implications.

New technological developments and the large amount of databases nowadays enable the use of profiling practices which collect, combine, analyse and automatically categorise data into groups. This automatic categorisation and identification of individuals' data enables business and governmental entities to classify individuals into certain profiles. Although such resulting profiles can help business entities to identify current or potential targets for their own benefits and decision-makings, profiling is likely to generate certain prejudicial treatments for the individuals, which may threaten their privacy and data protection rights, as personal and sensitive information may be revealed. By monitoring all individuals' activities, profiling enables different types of information to be merged to link to individuals' offline lives and thus to their physical identities. As such, profiling challenges the protection of individuals' fundamental rights and values and creates conditions for surveillance, manipulation, threats to individuals' autonomy, discrimination, deindividuation, stigmatisation, stereotyping and inaccuracy in the decision process. In addition, the asymmetries of knowledge and the unbalanced distribution of powers between the controllers and the data subjects are likely to affect the individuality of a person and generate concerns over their autonomy and their right to selfdetermination. Privacy and data protection are recognised as legal instruments which protect the rights of the individuals to preserve their freedom to develop their own unique identities and individuality within society. This thesis is limited to profiling practices operating in the private sector for commercial purposes. Extensive use has been made of primary sources, such as legislation, and other secondary materials, such as journals and textbooks, for the preparation of this thesis. The findings of this thesis argue that profiling contradicts the idea of transparency and the selfdeterminatory (controllable) nature of data protection legislation, and thus of the GDPR. In practice, the law is ineffective to safeguard the protection of individuals' fundamental rights to privacy and data protection in the context of profiling: consent as a control mechanism has proven to be a pseudo-right under which individuals are giving their consent mechanically and unconsciously and thus it cannot be considered as the freely given, specific, informed and unambiguous indication of the individuals' wishes; the de-individuated character of profiling makes individuals who have no interest either in exercising such control or to live a self-determined existence, and what is more, individuals are neither aware, nor able to be aware, that they are in this situation; profiling, as a Panopticon, corrects and controls individuals' behaviour without allowing individuals to freely exercise control over their data in order to make their own autonomous decisions and choices. It is, therefore, my opinion that the exercise of control should be left to the legislator rather than to the individuals themselves. Profiling constitutes a humanitarian issue and must be governed within the sphere of humanitarian law, as a stand-alone law of profiling in combination with fundamental human rights-based law.

Coastal regions are some of the most exposed to environmental hazards, yet the coast is the preferred settlement site for a high percentage of the global population, and most major global cities are located on or near the coast. This research adopts a predominantly anthropocentric approach to the analysis of coastal risk and resilience. This centres on the pervasive hazards of coastal flooding and erosion. Coastal management decision-making practices are shown to be reliant on access to current and accurate information. However, constraints have been imposed on information flows between scientists, policy makers and practitioners, due to a lack of awareness and utilisation of available data sources. This research seeks to tackle this issue in evaluating how innovations in the use of data and analytics can be applied to further the application of science within decision-making processes related to coastal risk adaptation. In achieving this aim a range of research methodologies have been employed and the progression of topics covered mark a shift from themes of risk to resilience. The work focuses on a case study region of East Anglia, UK, benefiting from the input of a partner organisation, responsible for the region’s coasts: Coastal Partnership East. An initial review revealed how data can be utilised effectively within coastal decision-making practices, highlighting scope for application of advanced Big Data techniques to the analysis of coastal datasets. The process of risk evaluation has been examined in detail, and the range of possibilities afforded by open source coastal datasets were revealed. Subsequently, open source coastal terrain and bathymetric, point cloud datasets were identified for 14 sites within the case study area. These were then utilised within a practical application of a geomorphological change detection (GCD) method. This revealed how analysis of high spatial and temporal resolution point cloud data can accurately reveal and quantify physical coastal impacts. Additionally, the research reveals how data innovations can facilitate adaptation through insurance; more specifically how the use of empirical evidence in pricing of coastal flood insurance can result in both communication and distribution of risk. The various strands of knowledge generated throughout this study reveal how an extensive range of data types, sources, and advanced forms of analysis, can together allow coastal resilience assessments to be founded on empirical evidence. This research serves to demonstrate how the application of advanced data-driven analytical processes can reduce levels of uncertainty and subjectivity inherent within current coastal environmental management practices. Adoption of methods presented within this research could further the possibilities for sustainable and resilient management of the incredibly valuable environmental resource which is the coast.

This thesis introduces the concept data-driven service systems (DDSS) to clarify the increasing adoption of big data and analytics for changing and improving the service systems where a configuration of entities (e.g. companies, customers, competitors, and diverse stakeholders) performs service exchanges and resource integration to co-create value. Grounded in a value co-creation perspective (e.g. Vargo and Lusch 2004, 2008; Gronroos 2008, 2011), this thesis proposes a DDSS framework. The framework can serve as a set of theoretical constructs, a strategic tool, and an IT prototype for researchers and practitioners to specify, design, implement, and evaluate the use of data in promoting positive system transformation. This thesis includes three prepared journal manuscripts representing three applications that evaluate the utility of the DDSS framework. The first and second paper were conducted to examining dialogue data collected from Twitter customer care platform for addressing the research issue of service recovery. The first paper aims to provide a better understanding of customer complaint management in the dynamic service environment and uncover important activities and contexts that influence customer satisfaction. The second paper theorised a data analytical model for mining service recovery dialogues and advanced the dialogue analysis that used to be done by qualitative data analysis methods. The dialogue-mining model facilitates text mining and process mining to investigate three dimensions of dialogue including linguistic and semantic, process and relationship, and thus allows researchers to capture more insightful knowledge from a vast volume of dialogue data. Finally, Twitter dialogue data was further applied to investigate the issue of corporate social innovation in the third paper. This paper demonstrated a data-driven approach to extract and internalise stakeholder knowledge embedded in dialogues and thus, indicate opportunities for social innovation. For each paper, novel data analytical approaches were developed to analyse unstructured data that comprises 95% of big data. In particular, text mining was used to automate information extraction in unstructured dialogue data based on specific domain knowledge (e.g. ontologies, dictionaries). In this way, text-mining approaches can provide contributions beyond the methodological and shed light into focal research domains.

The Hadoop framework has become the most deployed platform for processing Big Data. Despite its advantages, Hadoop s infrastructure is still deployed within the secured network perimeter because the framework lacks adequate inherent security mechanisms against various security threats. However, this approach is not sufficient for providing adequate security layer against attacks such as Distributed Denial of Service. Furthermore, current work to secure Hadoop s infrastructure against DDoS attacks is unable to provide a distributed node-level detection mechanism. This thesis presents a software agent-based framework that allows distributed, real-time intelligent monitoring and detection of DDoS attack at Hadoop s node-level. The agent s cognitive system is ingrained with cumulative sum statistical technique to analyse network utilisation and average server load and detect attacks from these measurements. The framework is a multi-agent architecture with transducer agents that interface with each Hadoop node to provide real-time detection mechanism. Moreover, the agents contextualise their beliefs by training themselves with the contextual information of each node and monitor the activities of the node to differentiate between normal and anomalous behaviours. In the experiments, the framework was exposed to TCP SYN and UDP flooding attacks during a legitimate MapReduce job on the Hadoop testbed. The experimental results were evaluated regarding performance metrics such as false-positive ratio, false-negative ratio and response time to attack. The results show that UDP and TCP SYN flooding attacks can be detected and confirmed on multiple nodes in nineteen seconds with 5.56% false-positive ration, 7.70% false-negative ratio and 91.5% success rate of detection. The results represent an improvement compared to the state-of the-art.

Over the last decade considerable interest has been generated into the use of analytical methods in organisations. Along with this, many have reported a significant gap between organisational demand for analytical-trained staff, and the number of potential recruits qualified for such roles. This interest is of high relevance to the operational research discipline, both in terms of raising the profile of the field, as well as in the teaching and training of graduates to fill these roles. However, what is less clear, is the extent to which operational research teaching in universities, or indeed teaching on the various courses labelled as analytics , are offering a curriculum that can prepare graduates for these roles. It is within this space that this research is positioned, specifically seeking to analyse the suitability of current provisions, limited to master s education in UK universities, and to make recommendations on how curricula may be developed. To do so, a mixed methods research design, in the pragmatic tradition, is presented. This includes a variety of research instruments. Firstly, a computational literature review is presented on analytics, assessing (amongst other things) the amount of research into analytics from a range of disciplines. Secondly, a historical analysis is performed of the literature regarding elements that can be seen as the pre-cursor of analytics, such as management information systems, decision support systems and business intelligence. Thirdly, an analysis of job adverts is included, utilising an online topic model and correlations analyses. Fourthly, online materials from UK universities concerning relevant degrees are analysed using a bagged support vector classifier and a bespoke module analysis algorithm. Finally, interviews with both potential employers of graduates, and also academics involved in analytics courses, are presented. The results of these separate analyses are synthesised and contrasted. The outcome of this is an assessment of the current state of the market, some reflections on the role operational research make have, and a framework for the development of analytics curricula. The principal contribution of this work is practical; providing tangible recommendations on curricula design and development, as well as to the operational research community in general in respect to how it may react to the growth of analytics. Additional contributions are made in respect to methodology, with a novel, mixed-method approach employed, and to theory, with insights as to the nature of how trends develop in both the jobs market and in academia. It is hoped that the insights here, may be of value to course designers seeking to react to similar trends in a wide range of disciplines and fields.

The aim of this research is to test the feasibility of using remote sensing-based information products and services to support the planning, monitoring and evaluation of recovery after disaster. The thesis begins by outlining the process of post-disaster recovery, what it entails and who is involved. The data and information needs at different stages of the disaster cycle are introduced and the importance of monitoring and evaluating post-disaster recovery is discussed. The literature review introduces the high-spatial-resolution remote sensing market and the technology focusing on current sensors’ capabilities. This is followed by a review of previous attempts to measure post-disaster recovery by practitioners and academics. At the end of the chapter a list of recovery indicators, suitable for remote sensing analysis, are presented and assessed through a user needs survey. In chapter 3, the six recovery categories and thirteen indicators identified in the literature review form a framework for the retrospective analysis of recovery in Thailand and Pakistan. A selection of results is presented to demonstrate the usefulness of remote sensing as a recovery monitoring tool. To assess its reliability, the results from the satellite image analysis are triangulated against narratives and datasets acquired on the ground. The next two chapters describe work done whilst providing real-time support to two humanitarian agencies operating in Port-au-Prince one-and-a-half years after the 2010 Haiti earthquake. Chapter 4 describes how geospatial tools were used to support a British Red Cross integrated reconstruction project for 500 households living in an informal settlement. The chapter describes how geospatial tools were used as a rapid assessment tool, and to support cadastral and enumeration mapping and the community participatory process. While previous chapters focus on the manual analysis of satellite imagery, chapter 5 reports how semi-automatic analyses of satellite imagery were used to support UN-Habitat by monitoring a planned camp and large-scale instances of spontaneous settlement. The conclusion to the thesis summarises the key lessons learnt from the retrospective analysis of recovery in Thailand and Pakistan and the real-time application in Haiti. Recommendations are then made on how to effectively use remote sensing in support of post-disaster recovery focussing on what to measure, when and how. Recognising that a mixed-method approach can best monitor recovery, recommendations are also made on how to integrate remote sensing with existing tools.

In recent years, modern prediction models have evolved to include behavioural data such as user-generated search query data that capture market sentiment and reach beyond the grasp of established macroeconomic indicators. These applications had considerable success in predicting a wide range of economic phenomena with the assumption that internet interaction behaviour resembles probable offline behaviour. Despite the considerable success of this approach, the existing literature argues for the continuous validation of search query keywords and its probable meaning over time to avoid spurious and biased results. Although recent literature attempted to bridge the keyword validation gap, this line of research is still in its infancy. This thesis sets out to examine the validity of web search intention to serve as a “pure” demand proxy for direct real estate market prediction in the United Kingdom. More specifically, it is directed towards constructing web search indices to explore: (i) the extent to which an individual’s true real estate orientated intentions manifest themselves in their web search behaviour and (ii) the magnitude to which real-time information adds value towards the prediction of illiquid asset classes. In doing so, a conceptual framework is produced, which outlines the logic and importance associated with intention specific web search in the digital age, as well as its relation to real estate demand. The empirical findings suggest that intention specific keyword development might be of little importance for aggregate housing and office market forecasts in the United Kingdom. On the contrary, it seems that the viability of intention specific web search keyword development increases when it is directed at a specific regional market. The overall thesis narrative introduces a new way of thinking about web search in the context of economic demand and draws from a variety of principles and methodologies to establish an avenue from which future research can be conducted.

Saltmarshes provide a diverse range of ecosystem services and are protected under a number of international designations. Nevertheless they are generally declining in extent in the United Kingdom and North West Europe. The drivers of this decline are complex and poorly understood. When considering mitigation and management for future ecosystem service provision it will be important to understand why, where, and to what extent decline is likely to occur. Few studies have attempted to forecast saltmarsh morphodynamics at a system level over decadal time scales. There is no synthesis of existing knowledge available for specific site predictions nor is there a formalised framework for individual site assessment and management. This project evaluates the extent to which machine learning model approaches (boosted regression trees, neural networks and Bayesian networks) can facilitate synthesis of information and prediction of decadal-scale morphological tendencies of saltmarshes. Importantly, data-driven predictions are independent of the assumptions underlying physically-based models, and therefore offer an additional opportunity to crossvalidate between two paradigms. Marsh margins and interiors are both considered but are treated separately since they are regarded as being sensitive to different process suites. The study therefore identifies factors likely to control morphological trajectories and develops geospatial methodologies to derive proxy measures relating to controls or processes. These metrics are developed at a high spatial density in the order of tens of metres allowing for the resolution of fine-scale behavioural differences. Conventional statistical approaches, as have been previously adopted, are applied to the dataset to assess consistency with previous findings, with some agreement being found. The data are subsequently used to train and compare three types of machine learning model. Boosted regression trees outperform the other two methods in this context. The resulting models are able to explain more than 95% of the variance in marginal changes and 91% for internal dynamics. Models are selected based on validation performance and are then queried with realistic future scenarios which represent altered input conditions that may arise as a consequence of future environmental change. Responses to these scenarios are evaluated, suggesting system sensitivity to all scenarios tested and offering a high degree of spatial detail in responses. While mechanistic interpretation of some responses is challenging, process-based justifications are offered for many of the observed behaviours, providing confidence that the results are realistic. The work demonstrates a potentially powerful alternative (and complement) to current morphodynamic models that can be applied over large areas with relative ease, compared to numerical implementations. Powerful analyses with broad scope are now available to the field of coastal geomorphology through the combination of spatial data streams and machine learning. Such methods are shown to be of great potential value in support of applied management and monitoring interventions.

Modern applications are increasingly backed by large-scale data centres. Systems software in these data centre environments, however, faces substantial challenges: the lack of uniform resource abstractions makes sharing and resource management inefficient, infrastructure software lacks end-to-end access control mechanisms, and work placement ignores the effects of hardware heterogeneity and workload interference. In this dissertation, I argue that uniform, clean-slate operating system (OS) abstractions designed to support distributed systems can make data centres more efficient and secure. I present a novel distributed operating system for data centres, focusing on two OS components: the abstractions for resource naming, management and protection, and the scheduling of work to compute resources. First, I introduce a reference model for a decentralised, distributed data centre OS, based on pervasive distributed objects and inspired by concepts in classic 1980s distributed OSes. Translucent abstractions free users from having to understand implementation details, but enable introspection for performance optimisation. Fine-grained access control is supported by combining storable, communicable identifier capabilities, and context-dependent, ephemeral handle capabilities. Finally, multi-phase I/O requests implement optimistically concurrent access to objects while supporting diverse application-level consistency policies. Second, I present the DIOS operating system, an implementation of my model as an extension to Linux. The DIOS system call API is centred around distributed objects, globally resolvable names, and translucent references that carry context-sensitive object meta-data. I illustrate how these concepts support distributed applications, and evaluate the performance of DIOS in microbenchmarks and a data-intensive MapReduce application. I find that it offers improved, finegrained isolation of resources, while permitting flexible sharing. Third, I present the Firmament cluster scheduler, which generalises prior work on scheduling via minimum-cost flow optimisation. Firmament can flexibly express many scheduling policies using pluggable cost models; it makes high-quality placement decisions based on fine-grained information about tasks and resources; and it scales the flow-based scheduling approach to very large clusters. In two case studies, I show that Firmament supports policies that reduce colocation interference between tasks and that it successfully exploits flexibility in the workload to improve the energy efficiency of a heterogeneous cluster. Moreover, my evaluation shows that Firmament scales the minimum-cost flow optimisation to clusters of tens of thousands of machines while still making sub-second placement decisions.

Survey research plays a significant role in the way psychologists investigate key relationships which impact human behaviours—and which inform us about undercurrents of a population. Samples are generally taken with the primary function of being able to make inferences which can be generalised to the target population—however, historically the field has consistently relied on small and niche datasets which are not truly representative of the general population. As a consequence, there is an abundance of published research which lacks ecological validity. The alternative approach is to collect larger amounts of data—this approach is extremely costly and in most instances impractical for the researcher. I have termed this conundrum, the cost-insight trade-off, which has traditionally exasperated psychologists. To address this dilemma, I conducted three studies using two alternative methods. Study 1 investigated the relationship between social status and international friendships at a micro and macro level. The building social status hypothesis states that higher social status individuals would reach out more to people and have more international friendships than their poorer counterparts. In contrast, the restrictive social status hypothesis states the higher social status individuals would be reclusive and have fewer international friendships than their poorer counterparts. Findings at both the micro (N = 857; U.S. participants) and macro levels (approximately 50 billion friendships across 187 countries) were in alignment with the restrictive social status hypothesis. Investigating this relationship at this large a scope would not have been possible without utilising Facebook Data—furthermore, for most research projects collecting data at this scale is both too costly and impractical. Study 2 aims to address the limitation of study 1. In this light, a new alternative method, the Survey Forecasting Method, is introduced and used to demonstrate creative capability of combining the latest technology, machine learning techniques and big data (i.e. Twitter). The findings were proof positive that a data collection of only 1,000 participants (at minimum) can be transformed into the power of having a dataset of several hundred thousand participants. In other words, the findings suggest that it is possible to efficiently and effectively forecast scores for potentially millions of people, without them having to complete a single survey. This is a significant step towards developing an alternative survey method; however, the method has only been applied to the Big Five & NEO-IPIP personality traits. Study 3 provides further evidence for the Survey Forecasting Method as a viable alternative to traditional sampling methods. The study examined the relationship between entrepreneurs’ self-efficacy, fear of failure, and well-being at two levels: (a) self-report and forecasted individual level, and (b) forecasted state level (across all 50 U.S. states). Findings show there are differences between each level which provides insights into effects and potential mechanisms which would not potentially be found using traditional “silo’d” methods. The primary aim of this thesis is to provide a viable alternative method to conducting survey research—which allows the researcher to gain deeper insights into the population at less cost and time. Furthermore, this alternative method addresses poor data representativeness. Limitations are addressed and future directions to improve its capability and robustness as a viable survey research methodology are provided.

Breast cancer is a clinically and molecularly heterogeneous disease displaying distinct therapeutic responses. Although recent studies have explored the genomic and transcriptomic landscapes of breast cancer, the epigenetic architecture has received less attention. To address this, an optimised Reduced Representation Bisulfite Sequencing protocol was performed on 1482 primary breast tumours (and 237 matched adjacent normal tissues). This constitutes the largest breast cancer methylome yet, and this thesis describes the bioinformatics and statistical analysis of this study. Noticeable epigenetic drift (both gain and loss of homogeneous DNA methylation patterns) was observed in breast tumours when compared to normal tissues, with markedly higher differences in late replicating genomic regions. The extent of epigenetic drift was also found to be highly heterogeneous between the breast tumours and was sharply correlated with the tumour’s mitotic index, indicating that epigenetic drift is largely a consequence of the accumulation of passive cell division related errors. A novel algorithm called DMARC (Directed Methylation Altered Regions in Cancer) was developed that utilised the tumour-specific drift rates to discriminate between methylation alterations attained as a consequence of stochastic cell division errors (background) and those reflecting a more instructive biological process (directed). Directed methylation alterations were significantly enriched for gene expression changes in breast cancer, compared to background alterations. Characterising these methylation aberrations with gene expression led to the identification of breast cancer subtype-specific epigenetic genes with consequences on transcription and prognosis. Cancer genes may be deregulated by multiple mechanisms. By integrating with existing copy number and gene expression profiles for these tumours, DNA methylation alterations were revealed as the predominant mechanism correlated with differentially expressed genes in breast cancer. The crucial role of DNA methylation as a mechanism to target the silencing of specific genes within copy number amplifications is also explored which led to the identification of a putative tumour suppressor gene, THSZ2. Finally, the first genome-wide assessment of epigenomic evolution in breast cancer is conducted. Both, the level of intratumoural heterogeneity, and the extent of epiallelic burden were found to be prognostic, and revealed an extraordinary distinction in the role of epiclonal dynamics in different breast cancer subtypes. Collectively, the results presented in this thesis have shed light on the somatic DNA methylation basis of inter-patient as well as intra-tumour heterogeneity in breast cancer. This complements our genetic knowledge of the disease, and will help move us towards tailoring treatments to the patient's molecular profile.

Gradient descent methods and especially their stochastic variants have become highly popular in the last decade due to their efficiency on big data optimization problems. In this thesis we present the development of data sampling strategies for these methods. In the first four chapters we focus on four views on the sampling for convex problems, developing and analyzing new state-of-the-art methods using non-standard data sampling strategies. Finally, in the last chapter we present a more flexible framework, which generalizes to more problems as well as more sampling rules. In the first chapter we propose an adaptive variant of stochastic dual coordinate ascent (SDCA) for solving the regularized empirical risk minimization (ERM) problem. Our modification consists in allowing the method to adaptively change the probability distribution over the dual variables throughout the iterative process. AdaSDCA achieves a provably better complexity bound than SDCA with the best fixed probability distribution, known as importance sampling. However, it is of a theoretical character as it is expensive to implement. We also propose AdaSDCA+: a practical variant which in our experiments outperforms existing non-adaptive methods. In the second chapter we extend the dual-free analysis of SDCA, to arbitrary mini-batching schemes. Our method is able to better utilize the information in the data defining the ERM problem. For convex loss functions, our complexity results match those of QUARTZ, which is a primal-dual method also allowing for arbitrary mini-batching schemes. The advantage of a dual-free analysis comes from the fact that it guarantees convergence even for non-convex loss functions, as long as the average loss is convex. We illustrate through experiments the utility of being able to design arbitrary mini-batching schemes. In the third chapter we study importance sampling of minibatches. Minibatching is a well studied and highly popular technique in supervised learning, used by practitioners due to its ability to accelerate training through better utilization of parallel processing power and reduction of stochastic variance. Another popular technique is importance sampling { a strategy for preferential sampling of more important examples also capable of accelerating the training process. However, despite considerable effort by the community in these areas, and due to the inherent technical difficulty of the problem, there is no existing work combining the power of importance sampling with the strength of minibatching. In this chapter we propose the first importance sampling for minibatches and give simple and rigorous complexity analysis of its performance. We illustrate on synthetic problems that for training data of certain properties, our sampling can lead to several orders of magnitude improvement in training time. We then test the new sampling on several popular datasets, and show that the improvement can reach an order of magnitude. In the fourth chapter we ask whether randomized coordinate descent (RCD) methods should be applied to the ERM problem or rather to its dual. When the number of examples (n) is much larger than the number of features (d), a common strategy is to apply RCD to the dual problem. On the other hand, when the number of features is much larger than the number of examples, it makes sense to apply RCD directly to the primal problem. In this paper we provide the first joint study of these two approaches when applied to L2-regularized ERM. First, we show through a rigorous analysis that for dense data, the above intuition is precisely correct. However, we find that for sparse and structured data, primal RCD can significantly outperform dual RCD even if d ≪ n, and vice versa, dual RCD can be much faster than primal RCD even if n ≫ d. Moreover, we show that, surprisingly, a single sampling strategy minimizes both the (bound on the) number of iterations and the overall expected complexity of RCD. Note that the latter complexity measure also takes into account the average cost of the iterations, which depends on the structure and sparsity of the data, and on the sampling strategy employed. We confirm our theoretical predictions using extensive experiments with both synthetic and real data sets. In the last chapter we introduce two novel generalizations of the theory for gradient descent type methods in the proximal setting. Firstly, we introduce the proportion function, which we further use to analyze all the known block-selection rules for coordinate descent methods under a single framework. This framework includes randomized methods with uniform, non-uniform or even adaptive sampling strategies, as well as deterministic methods with batch, greedy or cyclic selection rules. We additionally introduce a novel block selection technique called greedy minibatches, for which we provide competitive convergence guarantees. Secondly, the whole theory of strongly-convex optimization was recently generalized to a specific class of non-convex functions satisfying the so-called Polyak- Lojasiewicz condition. To mirror this generalization in the weakly convex case, we introduce the Weak Polyak- Lojasiewicz condition, using which we give global convergence guarantees for a class of non-convex functions previously not considered in theory. Additionally, we give local convergence guarantees for an even larger class of non-convex functions satisfying only a certain smoothness assumption. By combining the two above mentioned generalizations we recover the state-of-the-art convergence guarantees for a large class of previously known methods and setups as special cases of our framework. Also, we provide new guarantees for many previously not considered combinations of methods and setups, as well as a huge class of novel non-convex objectives. The flexibility of our approach offers a lot of potential for future research, as any new block selection procedure will have a convergence guarantee for all objectives considered in our framework, while any new objective analyzed under our approach will have a whole fleet of block selection rules with convergence guarantees readily available.

This thesis addresses the question of how data protection law should respond to the challenges arising from the ever-increasing prevalence of big data. The investigation is conducted with the case study of online behavioural advertising (OBA) and within the EU data protection legal framework, especially the General Data Protection Regulation (GDPR). It is argued that data protection law should respond to the big data challenges by leveraging the regulatory options that are either already in place in the current legal regime or potentially available to policymakers. With the highly complex, powerful and opaque OBA network, in both technical and economic terms, the use of big data may pose fundamental threats to certain individualistic, collective or societal values. Despite a limited number of economic benefits such as free access to online services and the growth of the digital market, the latent risks of OBA call for an effective regulatory regime on big data. While the EU's GDPR represents the latest and most comprehensive legal framework regulating the use of personal data, it has still fallen short on certain important aspects. The regulatory model characterised by individualised consent and the necessity test remains insufficient in fully protecting data subjects as autonomous persons, consumers and citizens in the context of OBA. There is thus a pressing need for policymakers to review their regulatory toolbox in the light of the potential threats. On the one hand, it is necessary to reconsider the possibilities to blacklist or whitelist certain data uses with mechanisms that are either in place in the legal framework or can be introduced additionally. On the other hand, it is also necessary to realise the full range of policy options that can be adopted to assist individuals in making informed decisions in the age of big data.

Integrated Pest Management (IPM) has long been promoted as a means of reducing reliance on pesticide inputs as compared to conventional farming systems. Reduced pesticide application could be beneficial due to the links between intensive pesticide use and negative impacts upon biodiversity and human health as well as the development of pesticide resistance. Work assessing the potential of IPM in cereal production is currently limited, however, and previous findings have generally covered the subject from the perspective of either field trial data or social science studies of farmer behaviour. This thesis attempts to help to address this knowledge gap by providing a more holistic assessment of IPM in Scottish spring barley production (selected because of its dominance in Scotland’s arable production systems), in relation to three of its most damaging fungal pathogens: Rhynchosporium commune, Blumeria graminis f.sp. hordei, and Ramularia collo-cygni. Several IPM techniques of potential relevance to the sector were identified, and the prospects of three in particular – crop rotation, varietal disease resistance, and forecasting disease pressure – were assessed in several ways. Preliminary analysis of experimental field trial data collected from 2011 – 2014 across Scotland found that the majority of spring barley trials in this period (65%) did not show a statistically significant impact of fungicide treatment on yield, with the average yield increase due to fungicide application being 0.62 t/ha. This initial analysis was expanded upon using stepwise regressions of long-term (1996 – 2014) field trial data from the same dataset. Here, the difference between treated and untreated yields could be explained by disease resistance, average seasonal rainfall (whereby wetter seasons saw an increased impact of fungicide use on yield), and high combined disease severity. Stakeholder surveying provided information about current practice and attitudes towards the selected IPM techniques amongst a group of 43 Scottish spring barley farmers and 36 agronomists. Stakeholders were broadly open to taking up IPM measures on farm; sowing of disease resistant varieties was most frequently selected as the best technique in terms of both practicality and cost, though individual preference varied. However, a disparity was seen between farmer perception of their uptake of IPM and actual, self-reported uptake for both varietal disease resistance and rotation. Farmers and agronomists also overestimated the impact of fungicide use as compared with the field trials results – the majority of stakeholders believed fungicide treatment to increase yields by 1 - 2 t/ha, while the majority of 2011 – 2014 field trials had a yield difference of under 1 t/ha. The reasons behind these differences between perception and practice are not currently known. Finally, an annual survey of commercial crops, gathered from 552 farms across Scotland (from 2009 – 2015), highlighted two gaps where IPM practice could be improved upon. Firstly, relatively few of the varieties listed in the commercial crops database were highly resistant to the three diseases – 26.1% were highly resistant to Ramularia, 14.2% to Rhynchosporium, and 58.1% to mildew. Secondly, 71% of the farms included in the database had planted barley in at least two consecutive seasons, indicating that crop rotation practices could be improved. The overarching finding of this project is that there is scope for IPM uptake to be improved upon and fungicide use to be reduced while maintaining high levels of yield in Scottish spring barley production. Incorporating experimental field data, stakeholder surveying, and commercial practice data offered a unique view into the potential for IPM in this sector, and provided insights which could not have been gained through the lens of a single discipline.

Data design, collection, and use are increasingly part of public life in the United Kingdom. Policy-makers in particular are engaging with these kinds of data technologies to increase efficiencies and cut costs in public service provision. Data technologies are subsequently increasingly important in mediating the processes and relationships of communities, citizens, and government. This thesis aims to examine how publics and policy-makers can engage with emerging data technologies in the context of local and national government in the UK. The objectives are as follows. First, to summarise and evaluate the current research and state of government public engagement with data technologies. Second, to scrutinise the current use of data technologies in government through qualitative semi-structured interviews and other qualitative methods. And third, to examine the potential for new forms of public data engagement in government through qualitative workshops and analysis of public views on government and data technologies. This thesis uses a mixed methods approach, which is primarily qualitative, to explore these objectives including a narrative literature review, ethnography, semi-structured interviews, story completion exercises, focus groups and statistical modelling through difference mediation analysis. Key findings include, first, that current public engagement practice on government data use is often limited in topic and form. For example, small-group discussions on public views of privacy that have no clear line to impact policy change. Second, that data use in government is subject to political motivation that generally takes the form of narrative testing. And third that public engagement in the era of big data in government must focus both on means and ends. Government must move beyond a model of seeing 'the public' as a roadblock on the way to civic datafication and instead as an integral part of debating the components of ethical and socially relevant data practice.

Methods for uncertainty quantification (UQ) and mitigation in the electrical power system are very basic, Monte Carlo (MC) method and its meta methods are generally deployed in most applications, due to its simplicity and easy to be generalised. They are adequate for a traditional power system when the load is predictable, and generation is controllable. However, the large penetration of low carbon technologies, such as solar panels, electric vehicles, and energy storage, has necessitated the needs for more comprehensive approaches to uncertainty as these technologies introduce new sources of uncertainties with larger volume and diverse characteristics, understanding source and consequences of uncertainty becomes highly complex issues. Traditional methods assume that for a given system it has a unique uncertainty characteristic, hence deal with the uncertainty of the system as a single component in applications. However, this view is no longer applicable in the new context as it neglects the important underlying information associated with individual uncertainty components. Therefore, this thesis aims at: i) systematically developing UQ methodologies to identify, discriminate, and quantify different uncertainty components (forward UQ), and critically to model and trace the associated sources independently (inverse UQ) to deliver new uncertainty information, such as, how uncertainty components generated from its sources, how uncertainty components correlate with each other and how uncertainty components propagate through system aggregation; ii) applying the new uncertainty information to further improve a range of fundamental power system applications from Load Forecasting (LF) to Energy Management System (EMS).In the EMS application, the proposed forward UQ methods enable the development of a decentralised system that is able to tap into the new uncertainty information concerning the correlations between load pattern across individual households, the characteristics of uncertainty components and their propagation through aggregation. The decentralised EMS was able to achieve peak and uncertainty reduction by 18% and 45% accordingly at the grid level. In the LF application, this thesis developed inverse UQ through a deep learning model to directly build the connection between uncertainty components and its corresponding sources. For Load Forecasting on expectation (point LF) and probability (probabilistic LF) and witnessed 20%/12% performance improvement compared to the state-of-the-art, such as Support Vector Regression (SVR), Autoregressive Integrated Moving Average (ARIMA), and Multiple Linear Quantile Regression (MLQR).

Patient records offer a very useful resource that can be used for many purposes, including advancement of clinical research and development of future health initiatives. The number of resources that provide access to patient data generated within primary care settings has recently seen a significant increase. Medical databases, such as the Clinical Practice Research Datalink (CPRD), were established to make access to and usage of patient records in research more efficient. As patient data become more available to researchers, this has the potential to drive further advancement in the medical field. In interrogating such data, researchers normally use specific a priori defined hypotheses. Conversely, other unexplored, yet equally important, areas may exist beyond these structured hypotheses, and therefore, this practice may reduce the impact of the data due to inadvertently missing relevant hypotheses. Therefore, advanced methods to generate hypotheses are vital to open such unexplored areas for research. This should assist in increasing the effectiveness of studies in elucidating underlying disease processes and potential outcomes of healthcare interventions. There is a lot of interest in applying machine learning data mining techniques for discovering patterns and correlations in large-scale datasets. Analysis of patient record data using such techniques could improve our understanding of the information included in these records. However, the representation of the data, bags of diagnostic codes using medical terminology, is not amenable to directly apply existing machine learning or data mining tools. The size of the datasets can also become an obstacle to applying such tools. In addressing these issues, this thesis undertakes the challenge of developing strategies for data exploration and hypothesis formulation from such resources. This thesis describes novel strategies that map electronic patient records from CPRD datasets into appropriate and informative low-dimensional vector spaces. Considering the high dimensionality and complexity of the data, we characterise the problem through developing a method that allows us to determine a minimal subset of patients that can be used to adequately represent all the dataset. Then, the methodology involves using this subset and employing analysis of semantic similarity between patients, principal components analysis (PCA) and clustering in order to provide effective representation of patient records. This assists in identifying unusual and interesting patterns at the subgroup level in order to investigate the different risks associated with different groups. In the final stage, a two-hit based strategy, which incorporates a temporal dimension, is used to explore combinations of risk factors most associated with increased risk of disease. The usefulness of this methodology was demonstrated mainly in assessing risk factors related falls in older adults as an exemplar case study. Patients experiencing falls provided a particularly rich dataset for this study. It is a diagnosis that is relatively common in a population that has significant contact with general practitioners (older patients), making it a suitable case study to test the methodology. A fundamental outcome of this study was the wealth of data that could be produced from such analysis. The machine learning strategy for agnostic hypothesis formulation successfully identified new hypotheses for factors associated with falls. These hypotheses were then examined based on existing literature and expert review to determine whether such data driven methodologies can provide new insights into risk factors associated with falls. In conclusion, we suggest that the strategies developed in this work would make available patient record data more amenable to analysis through traditional data mining strategies as well as allow a much more straightforward environment for agnostic hypothesis generation in this area of research. Furthermore, there is no aspect of these strategies that restricts it to falls. The same methodology could equally be applied to many other disease states extracted from any medical database using terms from taxonomies or ontologies.

The main purpose of this thesis is to develop a data quality scorecard (DQS) that aligns the data quality needs of the Data warehouse stakeholder group with selected data quality dimensions. To comprehend the research domain, a general and systematic literature review (SLR) was carried out, after which the research scope was established. Using Design Science Research (DSR) as the methodology to structure the research, three iterations were carried out to achieve the research aim highlighted in this thesis. In the first iteration, as DSR was used as a paradigm, the artefact was build from the results of the general and systematic literature review conduct. A data quality scorecard (DQS) was conceptualised. The result of the SLR and the recommendations for designing an effective scorecard provided the input for the development of the DQS. Using a System Usability Scale (SUS), to validate the usability of the DQS, the results of the first iteration suggest that the DW stakeholders found the DQS useful. The second iteration was conducted to further evaluate the DQS through a run through in the FMCG domain and then conducting a semi-structured interview. The thematic analysis of the semi-structured interviews demonstrated that the stakeholder's participants' found the DQS to be transparent; an additional reporting tool; Integrates; easy to use; consistent; and increases confidence in the data. However, the timeliness data dimension was found to be redundant, necessitating a modification to the DQS. The third iteration was conducted with similar steps as the second iteration but with the modified DQS in the oil and gas domain. The results from the third iteration suggest that DQS is a useful tool that is easy to use on a daily basis. The research contributes to theory by demonstrating a novel approach to DQS design This was achieved by ensuring the design of the DQS aligns with the data quality concern areas of the DW stakeholders and the data quality dimensions. Further, this research lay a good foundation for the future by establishing a DQS model that can be used as a base for further development.