Search

Statistical methods, and in particular deep learning models, have achieved remarkable success in computer vision, speech recognition, and natural language processing due to the availability of powerful computational resources. Recently, neuroscience and healthcare have entered an exciting new age. Modern recording technologies, like Multi-Electrode Arrays (MEA) and electronic health records (EHR), offer unprecedented opportunities to explore neural systems and to improve health care. At the same time, they present extraordinary computational and statistical challenges. This Ph.D. dissertation presents knowledge we mined from neuroscience data and healthcare data. In the neuroscience data mining part, we propose a deep learning framework for MEA classification of mouse and human derived induced Pluripotent Stem Cell recordings. We also introduce a scalable Bayesian framework for inference of functional neural networks from MEA data. In the healthcare mining part, we first perform quantitative analysis on early multiple organ failure (MOF) prediction with comprehensive machine learning (ML) configurations, including data preprocessing (missing value treatment, label balancing, feature scaling), feature selection,classifier choice), and hyperparameter tuning. We introduce BERTSurv, a deep learning survival framework which applies Bidirectional Encoder Representations from Transformers (BERT) as a language representation model on unstructured clinical notes, for mortality prediction and survival analysis. We propose classifier-guided generative adversarial imputation networks (Classifier-GAIN) for MOF prediction, by incorporating both observed data and label information. Finally, we propose an end-to-end deep learning framework for chronic pain score assessment.In the end, we summarize the strengths, weaknesses, and implications of our work, and discuss future research directions.

Monocular depth estimation, as one of the fundamental tasks of computer vision, plays a crucial role in three-dimensional (3D) scene understanding and perception. Usually, deep learning methods recover monocular depth maps using continuous regression manners by minimizing the errors between the ground-truth depth and the predicted depth. However, fine depth features may not be fully captured through layer-by-layer coding, which is prone to low spatial resolution depth maps and insufficient details. Furthermore, it usually converges slowly and suffers from unsatisfactory results. To tackle these issues, we propose a novel model, named context-based ordinal regression network (CORNet), to reconstruct monocular depth maps in the ordinal regression manner with context information in this paper. Firstly, we put forward a novel context-based encoder with a feature transformation (FT) module to learn context information and details from inputs, and output multi-scale feature maps. Then, we design a boundary enhancement module (BEM) with a spatial attention mechanism following each operation of feature fusion, which captures boundary features in the scene to enhance the border depth. Finally, a feature optimization module (FOM) is designed to fuse and optimize the multi-scale features and boundary features to strengthen depth learning. What’s more, we introduce an ordinal weighted inference to predict depth maps from probabilities and discretization values. Experiments and results on two challenging datasets, KITTI and NYU Depth V2, demonstrate that our proposed CORNet can estimate monocular depth maps effectively and obtain superior performance in capturing geometric features over existing methods.

We propose a novel Bayesian Ordinal Regression approach for multiple criteria choice and ranking problems. It employs an additive value function model to represent indirect Decision Maker’s (DM’s) preferences in the form of pairwise comparisons of reference alternatives. By defining a likelihood for the provided preference information and specifying a prior of the preference model, we apply the Bayesian rule to derive a posterior distribution over a set of all potential value functions, not necessarily compatible ones. This distribution emphasizes the potential differences in the abilities of these models to reconstruct the DM’s pairwise comparisons. Hence a distinctive character of our approach consists of characterizing the uncertainty in consequence of applying indirect preference information. We also employ a Markov Chain Monte Carlo algorithm, called the Metropolis-Hastings method, to summarize the posterior distribution of the value function model and quantify the outcomes of robustness analysis in the form of stochastic acceptability indices. The proposed approach’s performance is investigated in a thorough experimental study involving real-world and artificially generated datasets.

Model-based market segmentation analyses often involve an ordinal dependent variable as ordinal responses are frequently collected in marketing research. In the Bayesian segmentation literature, there are models for an interval- or ratio-scaled dependent variable but there is not any general model for an ordinal dependent variable. In this manuscript, the authors propose a new Bayesian procedure to simultaneously perform segmentation and ordinal regression with variable selection within each derived segment. The procedure is robust to outliers and it also provides an option to include concomitant variables that allows the simultaneous profiling of the derived segments. The authors demonstrate that the practice of treating ordinal responses as interval- or ratio-scales to apply existing Bayesian segmentation procedures can lead to very misleading results and conclusions. Through simulation studies, the authors show that the proposed procedure outperforms several benchmark Bayesian segmentation models in parameter recovery, segment retention, and segment membership prediction for such data. Finally, they provide a commercial business customer satisfaction empirical application to illustrate the usefulness of the proposed model.

Ordinal regression predicts the objects’ labels that exhibit a natural ordering, which is vital to decision-making problems such as credit scoring and clinical diagnosis. In these problems, the ability to explain how the individual features and their interactions affect the decisions is as critical as model performance. Unfortunately, the existing ordinal regression models in the machine learning community aim at improving prediction accuracy rather than explore explainability. To achieve high accuracy while explaining the relationships between the features and the predictions, we propose a new method for ordinal regression problems, namely the Explainable Ordinal Factorization Model (XOFM). XOFM uses piecewise linear functions to approximate the shape functions of individual features, and renders the pairwise features interaction effects as heat-maps. The proposed XOFM captures the nonlinearity in the main effects and ensures the interaction effects’ same flexibility. Therefore, the underlying model yields comparable performance while remaining explainable by explicitly describing the main and interaction effects. To address the potential sparsity problem caused by discretizing the whole feature scale into several sub-intervals, XOFM integrates the Factorization Machines (FMs) to factorize the model parameters. Comprehensive experiments with benchmark real-world and synthetic datasets demonstrate that the proposed XOFM leads to state-of-the-art prediction performance while preserving an easy-to-understand explainability.

The purpose of this paper is to introduce a new distance metric learning algorithm for ordinal regression. Ordinal regression addresses the problem of predicting classes for which there is a natural ordering, but the real distances between classes are unknown. Since ordinal regression walks a fine line between standard regression and classification, it is a common pitfall to either apply a regression-like numerical treatment of variables or underrate the ordinal information applying nominal classification techniques. On a different note, distance metric learning is a discipline that has proven to be very useful when improving distance-based algorithms such as the nearest neighbors classifier. In addition, an appropriate distance can enhance the explainability of this model. In our study we propose an ordinal approach to learning a distance, called chain maximizing ordinal metric learning. It is based on the maximization of ordered sequences in local neighborhoods of the data. This approach takes into account all the ordinal information in the data without making use of any of the two extremes of classification or regression, and it is able to adapt to data for which the class separations are not clear. We also show how to extend the algorithm to learn in a non-linear setup using kernel functions. We have tested our algorithm on several ordinal regression problems, showing a high performance under the usual evaluation metrics in this domain. Results are verified through Bayesian non-parametric testing. Finally, we explore the capabilities of our algorithm in terms of explainability using the case-based reasoning approach. We show these capabilities empirically on two different datasets, experiencing significant improvements over the case-based reasoning with the traditional Euclidean nearest neighbors.

Driver mental fatigue leads to thousands of traffic accidents. The increasing quality and availability of low-cost electroencephalogram (EEG) systems offer possibilities for practical fatigue monitoring. However, non-data-driven methods, designed for practical, complex situations, usually rely on handcrafted data statistics of EEG signals. To reduce human involvement, we introduce a data-driven methodology for online mental fatigue detection: self-weight ordinal regression (SWORE). Reaction time (RT), referring to the length of time people take to react to an emergency, is widely considered an objective behavioral measure for mental fatigue state. Since regression methods are sensitive to extreme RTs, we propose an indirect RT estimation based on preferences to explore the relationship between EEG and RT, which generalizes to any scenario when an objective fatigue indicator is available. In particular, SWORE evaluates the noisy EEG signals from multiple channels in terms of two states: shaking state and steady state. Modeling the shaking state can discriminate the reliable channels from the uninformative ones, while modeling the steady state can suppress the task-nonrelevant fluctuation within each channel. In addition, an online generalized Bayesian moment matching (online GBMM) algorithm is proposed to online-calibrate SWORE efficiently per participant. Experimental results with 40 participants show that SWORE can maximally achieve consistent with RT, demonstrating the feasibility and adaptability of our proposed framework in practical mental fatigue estimation.

Objective The prevalence of social media for sharing personal thoughts makes it a viable platform for the assessment of suicide risk. However, deep learning models are not able to capture the diverse nature of linguistic choices and temporal patterns that can be exhibited by a suicidal user on social media and end up overfitting on specific cues that are not generally applicable. We propose Adversarial Suicide assessment Hierarchical Attention (ASHA), a hierarchical attention model that employs adversarial learning for improving the generalization ability of the model. Material and Methods We assess the suicide risk of a social media user across 5 levels of increasing severity of risk. ASHA leverages a transformer-based architecture to learn the semantic nature of social media posts and a temporal attention-based long short-term memory architecture for the sequential modeling of a user’s historical posts. We dynamically generate adversarial examples by adding perturbations to actual examples that can simulate the stochasticity in historical posts, thereby making the model robust. Results Through extensive experiments, we establish the face-value of ASHA and show that it significantly outperforms existing baselines, with the F1 score of 64%. This is a 2% and a 4% increase over the ContextBERT and ContextCNN baselines, respectively. Finally, we discuss the practical applicability and ethical aspects of our work pertaining to ASHA, as a human-in-the-loop framework. Discussion and Conclusions Adversarial samples can be helpful in capturing the diverse nature of suicidal ideation. Through ASHA, we hope to form a component in a larger human-in-the-loop infrastructure for suicide risk assessment on social media.

Many real-world datasets are labeled with natural orders, i.e., ordinal labels. Ordinal regression is a method to predict ordinal labels that finds a wide range of applications in data-rich domains, such as natural, health and social sciences. Most existing ordinal regression approaches work well for independent and identically distributed (IID) instances via formulating a single ordinal regression task. However, for heterogeneous non-IID instances with well-defined local geometric structures, e.g., subpopulation groups, multi-task learning (MTL) provides a promising framework to encode task (subgroup) relatedness, bridge data from all tasks, and simultaneously learn multiple related tasks in efforts to improve generalization performance. Even though MTL methods have been extensively studied, there is barely existing work investigating MTL for heterogeneous data with ordinal labels. We tackle this important problem via sparse and deep multi-task approaches. Specifically, we develop a regularized multi-task ordinal regression (MTOR) model for smaller datasets and a deep neural networks based MTOR model for large-scale datasets. We evaluate the performance using three real-world healthcare datasets with applications to multi-stage disease progression diagnosis. Our experiments indicate that the proposed MTOR models markedly improve the prediction performance comparing with single-task ordinal regression models., Comment: 21 pages, 3 figures

With the development of social media sites, user credit grading, which served as an important and fashionable problem, has attracted substantial attention from a slew of developers and operators of mobile applications. In particular, multi-grades of user credit aimed to achieve (1) anomaly detection and risk early warning and (2) personalized information and service recommendation for privileged users. The above two goals still remained as up-to-date challenges. To these ends, in this article, we propose a novel regression-based method. Technically speaking, we define three natural ordered categories including BlockList , GeneralList , and AllowList according to users’ registration and behavior information, which preserve both the global hierarchical relationship of user credit and the local coincident features of users, and hence formulate user credit grading as the ordinal regression problem. Our method is inspired by KDLOR ( kernel discriminant learning for ordinal regression ), which is an effective and efficient model to solve ordinal regression by mapping high-dimension samples to the discriminant region with supervised conditions. However, the performance of KDLOR is fragile to the extreme imbalanced distribution of users. To address this problem, we propose a robust sampling model to balance distribution and avoid overfit or underfit learning, which induces the triplet metric constraint to obtain hard negative samples that well represent the latent ordered class information. A step further, another salient problem lies in ambiguous samples that are noises or located in the classification boundary to impede optimized mapping and embedding. To this problem, we improve sampling by identifying and evading noises in triplets to obtain hard negative samples to enhance robustness and effectiveness for ordinal regression. We organized training and testing datasets for user credit grading by selecting limited items from real-life huge tables of users in the mobile application, which are used in similar problems; moreover, we theoretically and empirically demonstrate the advantages of the proposed model over established datasets.

Osteoarthritis (OA) is a common form of knee arthritis which causes significant disability and is threatening to plague patient’s quality of life. Although this chronic condition does not lead to fatality, still there exists no known cure for OA. Diagnosis of OA can be confirmed primarily based on radiographic findings. Being a progressive disease, early identification of OA is crucial for clinical interventions to curtail the OA degeneration. Kellgren-Lawrence (KL) grading system has been traditionally employed to assess the knee OA severity. Due to the recent advancements of deep learning in computer vision, more studies have employed deep neural network in automatically predicting KL grade from plain knee joint radiograph. However, these studies treat KL grading as a multi-class classification task and ignore the inherent ordinal nature within the KL grades. In this study, we propose an ordinal regression module for neural networks to treat KL grading as an ordinal regression task. Our module takes an input from neural network and produces 4 cut-points to partition the prediction space into 5 respective KL grades. The proposed model is optimized by a cumulative-link loss function. Performance of the model is evaluated against various notable neural networks and significant improvements on the knee OA KL grade prediction were demonstrated.

Many real-world data are labeled with natural orders, i.e., ordinal labels. Examples can be found in a wide variety of fields. Ordinal regression is a problem to predict ordinal labels for given patterns. There are specially developed ordinal regression methods to tackle this type of problems, but they are usually centralized. However, in some scenarios, data are collected distributedly by nodes of a network. For the purpose of privacy protection or due to some practical constraints, it is difficult or impossible to transmit the data to a fusion center for processing. Thus the centralized ordinal regression methods are inapplicable. In this paper, we formulate a distributed generalized ordered logit model for distributed ordinal regression. To estimate parameters in the model, a distributed constrained optimization formulation based on maximum likelihood methods is established. Then, we propose a projected gradient based algorithm to solve the optimization problem. We prove the consensus and the convergence of the proposed distributed algorithm. We also conduct numerical simulations on synthetic and real-world datasets. Simulation results show that the proposed distributed algorithm is comparable to the corresponding centralized algorithm. Even when the data label distribution among nodes is unbalanced, the proposed algorithm still has competitive performance.

Training models by minimizing surrogate loss functions with gradient-based algorithms is a standard approach in various vision tasks. This strategy often leads to suboptimal solutions due to the gap between the target evaluation metrics and surrogate loss functions. In this paper, we propose a framework to learn a surrogate loss function that approximates the evaluation metric with correlated gradients. We observe that the correlated gradients significantly benefit the gradient-based algorithms to improve the quality of solutions. We verify the effectiveness of our method in various tasks such as multi-class classification, ordinal regression, and pose estimation with three evaluation metrics and five datasets. Our extensive experiments showed that our method outperforms conventional loss functions and surrogate loss learning methods.

Pedestrians face difficulties when trying to cross a road in the urban environment since they are exposed to vehicular traffic. One of the measures to overcome the problems in the case of multilane roads with high traffic volumes is the construction of footbridges. However, pedestrians are not always in favor of footbridges for various reasons (e.g., excessive effort to use the footbridge, deviation from their scheduled route etc.). Thus, pedestrians often face a dilemma, whether to use the footbridge or the signalized crossing. In the framework of this paper an attempt has been made to examine the factors that influence the use of footbridges in the urban environment and the factors that influence the use of signalized crossings in the near area of the footbridge. The case study refers to a major arterial road with heavy traffic volumes in the city of Thessaloniki, Greece. A statistical model was developed, quantifying the impact of quantitative and qualitative factors, as well as pedestrians' social characteristics on the frequency of using the footbridge. The study area included two signalized crossings and the footbridge. Pedestrians can choose either the footbridge or the signalized intersections to cross the road. Counts were made concerning traffic volume, vehicle speed and pedestrian flow. A questionnaire- based survey including 130 interviewees was conducted among pedestrians referring to the three ways of crossing the road (2 level crossings and the footbridge). Almost half of the pedestrians (49%) stated that they never use the footbridge. Most of the responders (87%) consider the footbridge very safe even though they do not use it. In the analysis made in this paper, an ordinal regression model was developed that utilizes questionnaire survey data and field measurements. The aim of the ordinal regression model is to investigate the variables affecting the operation of the footbridge through measuring the frequency of choosing the footbridge to cross the road. The model indicated that pedestrians aged between 25-39 years old have less possibilities to cross using the footbridge. The odds ratio calculation revealed that interviewees who state that the footbridge is very easy to use are more likely to use the footbridge frequently compared to those who state that the footbridge is not at all or almost at all easy to use. The ordinal regression model predicted 0,2 times higher possibility to use the footbridge seldom when interviewees say that their distance from the footbridge affects a little or a lot their crossing point decision. Finally, amongst other important outcomes, the model revealed that interviewees spotted on the footbridge are more likely to use it regularly, thus using the footbridge is more a habit than a random act. Statistical models of this type may help researchers understand pedestrians' attitudes better and potentially contribute to a far better design of new infrastructures or better management of existing infrastructures, considering the user's point of view.

Canonical correlation analysis (CCA) is a typical and useful learning paradigm in big data analysis for capturing correlation across multiple views of the same objects. When dealing with data with additional ordinal information, traditional CCA suffers from poor performance due to ignoring the ordinal relationships within the data. Such data is becoming increasingly common, as either temporal or sequential information is often associated with the data collection process. To incorporate the ordinal information into the objective function of CCA, the so-called ordinal discriminative CCA has been presented in the literature. Although ordinal discriminative CCA can yield better ordinal regression results, its performance deteriorates when data is corrupted with noise and outliers, as it tends to smear the order information contained in class centers. To address this issue, in this article we construct a robust manifold-preserved ordinal discriminative correlation regression (rmODCR). The robustness is achieved by replacing the traditional ( l 2 -norm) class centers with l p -norm centers, where p is efficiently estimated according to the moments of the data distributions, as well as by incorporating the manifold distribution information of the data in the objective optimization. In addition, we further extend the robust manifold-preserved ordinal discriminative correlation regression to deep convolutional architectures. Extensive experimental evaluations have demonstrated the superiority of the proposed methods.

In this article, we propose a Bayesian Adaptive bridge regression for ordinal model. We developed a new hierarchical model for ordinal regression in the Bayesian adaptive bridge. We consider a fully Bayesian approach that yields a new algorithm with tractable full conditional posteriors. All of the results in real data and simulation application indicate that our method is effective and performs very good compared to other methods. We can also observe that the estimator parameters in our proposed method, compared with other methods, are very close to the true parameter values.

This paper targets for the ordinal regression/classification, which objective is to learn a rule to predict labels from a discrete but ordered set. For instance, the classification for medical diagnosis usually involves inherently ordered labels corresponding to the level of health risk. Previous multi-task classifiers on ordinal data often use several binary classification branches to compute a series of cumulative probabilities. However, these cumulative probabilities are not guaranteed to be monotonically decreasing. It also introduces a large number of hyper-parameters to be fine-tuned manually. This paper aims to eliminate or at least largely reduce the effects of those problems. We propose a simple yet efficient way to rephrase the output layer of the conventional deep neural network. Besides, in order to alleviate the effects of label noise in ordinal datasets, we propose a unimodal label regularization strategy. It also explicitly encourages the class predictions to distribute on nearby classes of ground truth. We show that our methods lead to the state-of-the-art accuracy on the medical diagnose task (e.g., Diabetic Retinopathy and Ultrasound Breast dataset) as well as the face age prediction (e.g., Adience face and MORPH Album II) with very little additional cost.

This paper demonstrates the use of two Bayesian statistical models to analyze single-event sonic boom exposure and human annoyance data from community response surveys. Each model is fit to data from a NASA pilot study. Unlike many community noise surveys, this study used a panel sample to collect multiple observations per participant instead of a single observation. Thus, a multilevel (also known as hierarchical or mixed-effects) model is used to account for the within-subject correlation in the panel sample data. This paper describes a multilevel logistic regression model and a multilevel ordinal regression model. The paper also proposes a method for calculating a summary dose-response curve from the multilevel models that represents the population. The two models' summary dose-response curves are visually similar. However, their estimates differ when calculating the noise dose at a fixed percent highly annoyed.

In scoring systems used to measure the endoscopic activity of ulcerative colitis, such as Mayo endoscopic score or Ulcerative Colitis Endoscopic Index Severity, levels increase with severity of the disease activity. Such relative ranking among the scores makes it an ordinal regression problem. On the other hand, most studies use categorical cross-entropy loss function to train deep learning models, which is not optimal for the ordinal regression problem. In this study, we propose a novel loss function, class distance weighted cross-entropy (CDW-CE), that respects the order of the classes and takes the distance of the classes into account in calculation of the cost. Experimental evaluations show that models trained with CDW-CE outperform the models trained with conventional categorical cross-entropy and other commonly used loss functions which are designed for the ordinal regression problems. In addition, the class activation maps of models trained with CDW-CE loss are more class-discriminative and they are found to be more reasonable by the domain experts.

Skew-Gaussian Processes (SkewGPs) extend the multivariate Unified Skew-Normal distributions over finite dimensional vectors to distribution over functions. SkewGPs are more general and flexible than Gaussian processes, as SkewGPs may also represent asymmetric distributions. In a recent contribution, we showed that SkewGP and probit likelihood are conjugate, which allows us to compute the exact posterior for non-parametric binary classification and preference learning. In this paper, we generalize previous results and we prove that SkewGP is conjugate with both the normal and affine probit likelihood, and more in general, with their product. This allows us to (i) handle classification, preference, numeric and ordinal regression, and mixed problems in a unified framework; (ii) derive closed-form expression for the corresponding posterior distributions. We show empirically that the proposed framework based on SkewGP provides better performance than Gaussian processes in active learning and Bayesian (constrained) optimization. These two tasks are fundamental for design of experiments and in Data Science.

Ordinal regression addresses the problem of predicting non-numerical ordered classes. It walks a fine line between standard regression and classification, and the problem is often addressed from one of these perspectives. This can lead to suboptimal results as the ordinal information in the data may not be properly exploited. In this work we propose a distance metric learning algorithm to handle ordinal regression. Our model aims at optimizing the number of ordered sequences in local neighborhoods of the data, so that the learned distance can then be used by a distance-based predictor and improve its performance in ordinal regression problems. We evaluate our algorithm on several ordinal regression datasets and show that it outperforms the current distance metric learning for ordinal regression proposals, as well as being competitive with respect to the state-of-the-art of ordinal regression. The current paper is an extended abstract for the work [1].

This editorial overview provides an introduction to the Suicide and Life-Threatening Behaviors Special Issue: "Analytic and Methodological Innovations for Suicide-Focused Research." We outline several challenges faced by modern suicidologists, such as the need to integrate different analytical and methodological techniques from other fields with the unique data problems in suicide research. Therefore, the overall aim of this issue was to provide up-to-date methodological and analytical guidelines, recommendations, and considerations when conducting suicide-focused research. The articles herein present this information in an accessible way for researchers/clinicians and do not require a comprehensive background in quantitative methods. We introduce the topics covered in this special issue, which include how to conduct power analyses using simulations, work with large data sets, use experimental therapeutics, and choose covariates, as well as open science considerations, decision-making models, ordinal regression, machine learning, network analysis, and measurement considerations. Many of the topics covered in this issue provide step-by-step walkthroughs using worked examples with the accompanied code in free statistical programs (i.e., R). It is our hope that these articles provide suicidologists with valuable information and strategies that can help overcome some of the past limitations of suicide research, and improve the methodological rigor of our field.

Many high-throughput genomic applications involve a large set of potential covariates and a response which is frequently measured on an ordinal scale, and it is crucial to identify which variables are truly associated with the response. Effectively controlling the false discovery rate (FDR) without sacrificing power has been a major challenge in variable selection research. This study reviews two existing variable selection frameworks, model-X knockoffs and a modified version of reference distribution variable selection (RDVS), both of which utilize artificial variables as benchmarks for decision making. Model-X knockoffs constructs a ‘knockoff’ variable for each covariate to mimic the covariance structure, while RDVS generates only one null variable and forms a reference distribution by performing multiple runs of model fitting. Herein, we describe how different importance measures for ordinal responses can be constructed that fit into these two selection frameworks, using either penalized regression or machine learning techniques. We compared these measures in terms of the FDR and power using simulated data. Moreover, we applied these two frameworks to high-throughput methylation data for identifying features associated with the progression from normal liver tissue to hepatocellular carcinoma to further compare and contrast their performances.

Monocular depth estimation poses a fundamental problem in many tasks. Although recent convolutional neural network-based methods can achieve high accuracy with very deep networks and complex architectures to exploit different cues and features, doing so not only increases the vulnerability of the model, but also increases the difficulty of convergence. Moreover, recent depth estimation methods for indoor environments are impractical for outdoor environments. In this work, we aim to develop a simple deep network structure to improve model effectiveness for depth estimation. We apply a dual attention module that can be inserted into any type of network to improve the power of representation, and additionally propose a training strategy which combines transfer learning and ordinal regression to improve training convergence. Even with a simple end-to-end encoder-decoder type of network architecture, we are able to achieve state-of-the-art performance on two of the biggest datasets for indoor and outdoor depth estimation: NYU Depth v2 and KITTI.

Some recent work treats age estimation as an ordinal ranking task and decomposes it into multiple binary classifications. However, a theoretical defect lies in this type of methods: the ignorance of possible contradictions in individual ranking results. In this paper, we partially embrace the decomposition idea and propose the Deep and Ordinal Ensemble Learning with Two Groups Classification (DOEL2groups) for age prediction. An important advantage of our approach is that it theoretically allows the prediction even when the contradictory cases occur. The proposed method is characterized by a deep and ordinal ensemble and a two-stage aggregation strategy. Specifically, we first set up the ensemble based on Convolutional Neural Network (CNN) techniques, while the ordinal relationship is implicitly constructed among its base learners. Each base learner will classify the target face into one of two specific age groups. After achieving probability predictions of different age groups, then we make aggregation by transforming them into counting value distributions of whole age classes and getting the final age estimation from their votes. Moreover, to further improve the estimation performance, we suggest to regard the age class at the boundary of original two age groups as another age group and this modified version is named the Deep and Ordinal Ensemble Learning with Three Groups Classification (DOEL3groups). Effectiveness of this new grouping scheme is validated in theory and practice. Finally, we evaluate the proposed two ensemble methods on controlled and wild aging databases, and both of them produce competitive results. Note that the DOEL3groups shows the state-of-the-art performance in most cases.

Ordinal regression (OR), also called ordinal classification, is a special multi-classification designed for problems with ordered classes. Imbalanced data hinders the performance of classification algorithms, especially for OR algorithms, as imbalanced class distributions often arise in OR problems. In this article, we address an active learning based solution for imbalanced OR problem. We propose an active learning algorithm for OR (AL-OR) to select the most informative samples from unlabeled samples, mark them and add them to the training set. Based on AL-OR, we put forward an improved active learning for imbalanced OR (IAL-IOR), which further adjust the sampling strategy of AL-OR dynamically to make the training data as valuable and balanced as possible. Recall rate for multi-classification and new mean absolute error are designed to measure the performance of the algorithms. To the best of our knowledge, our algorithm is the first algorithm for imbalanced OR in algorithm level. The experimental results show that the proposed algorithms have faster convergence and much better generalization ability than the classical methods and the state-of-the-art methods under the evaluation measurements for imbalance problems. In addition, we also proved the effectiveness of our algorithms by statistical analysis.

A novel algorithm to estimate instance-level future motion (FM) in a single image is proposed in this paper. First, the FM of an instance is defined with its direction, speed, and action classes. Then, a deep neural network, called FM-Net, is developed to determine the FM of the instance. More specifically, the multi-context pooling layer is proposed to exploit both object and global context features, and the cyclic ordinal regression scheme is developed using binary classifiers for effective FM classification. Also, the proposed FM-Net is trained in a semi-supervised domain adaptation setting to obtain reliable FM estimation results, even when a source domain in the training process and a target domain in the inference process are different. Extensive experimental results demonstrate that the proposed algorithm provides remarkable performance and thus can be used effectively for computer vision applications, including single object tracking, multiple object tracking, and crowd analysis. Furthermore, the FM dataset, collected from diverse sources and annotated manually, is released as a benchmark for single-image FM estimation.

An OSNR monitoring method assisted by support vector ordinal regression (SVOR) is proposed for digital coherent receivers. According to Godard's error of equalized signals obtained after polarization demultiplexing, the optical signal-to-noise ratio (OSNR) value could be effectively estimated by using SVOR. The effectiveness of the proposed OSNR monitoring method is verified via numerical simulations with 28-Gbaud PDM-QPSK, PDM-8PSK, PDM-8QAM, PDM-16QAM and PDM-32QAM signals. The simulation results show that the 100% accuracies for the five modulation formats can be obtained over a wide OSNR range. The feasibility is further demonstrated by proof-of-concept experiments in 28-Gbaud PDM-QPSK, PDM-8PSK, PDM-8QAM and PDM-16QAM systems. The errors of averaged values of estimated OSNR for all the signals are less than or equal to 0.2 dB.

We consider a predictive modelling problem, where the goal is to predict the absolute evaluation of an object on an ordinal scale, traditionally known as an ordinal regression problem. We present a framework that is capable of learning such a model while combining different types of information: absolute evaluations by experts and relative evaluations by novices. We propose and solve a linearly constrained convex optimization problem that takes both types of information into account, and is capable of attributing an ordinal label to a new object based on its features. We do this by relying on principles from machine learning and optimization theory, combined with ideas from information fusion. Experimental results demonstrate the enhanced performance of ordinal regression models when incorporating relative evaluations in the form of rankings.

We propose a family of algorithms, called EMOSOR, combining Evolutionary Multiple Objective Optimization with Stochastic Ordinal Regression. The proposed methods ask the Decision Maker (DM) to holistically compare, at regular intervals, a pair of solutions, and use the Monte Carlo simulation to construct a set of preference model instances compatible with such indirect and incomplete information. The specific variants of EMOSOR are distinguished by the following three aspects. Firstly, they make use of two different preference models, i.e., either an additive value function or a Chebyshev function. Secondly, they aggregate the acceptability indices derived from the stochastic analysis in various ways, and use thus constructed indicators or relations to sort the solutions obtained in each generation. Thirdly, they incorporate different active learning strategies for selecting pairs of solutions to be critically judged by the DM. The extensive computational experiments performed on a set of benchmark optimization problems reveal that EMOSOR is able to bias an evolutionary search towards a part of the Pareto front being the most relevant to the DM, outperforming in this regard the state-of-the-art interactive evolutionary hybrids. Moreover, we demonstrate that the performance of EMOSOR improves in case the forms of a preference model used by the method and the DM’s value system align. Furthermore, we discuss how vastly incorporation of different indicators based on the stochastic acceptability indices influences the quality of both the best constructed solution and an entire population. Finally, we demonstrate that our novel questioning strategies allow to reduce a number of interactions with the DM until a high-quality solution is constructed or, alternatively, to discover a better solution after the same number of interactions.

Joint modeling of the intensity of multiple facial action units (AUs) from face images is challenging due to the large number of AUs (30+) and their intensity levels (6). This is in part due to the lack of suitable models that can efficiently handle such a large number of outputs/classes simultaneously, but also due to the lack of suitable data the models on. For this reason, majority of the methods resort to independent classifiers for the AU intensity. This is suboptimal for at least two reasons: the facial appearance of some AUs changes depending on the intensity of other AUs, and some AUs co-occur more often than others. To this end, we propose the Copula regression approach for modeling multivariate ordinal variables. Our model accounts for $ordinal$ordinal structure in output variables and their $non$non-$linear$linear dependencies via copula functions modeled as cliques of a conditional random fields. The copula ordinal regression model achieves the joint learning and inference of intensities of multiple AUs, while being computationally tractable. We demonstrate the effectiveness of our approach on three challenging datasets of naturalistic facial expressions and we show that the estimation of target AU intensities improves especially in the case of (a) noisy image features, (b) head-pose variations and (c) imbalanced training data. Lastly, we show that the proposed approach consistently outperforms (i) independent modeling of AU intensities and (ii) the state-of-the-art approach for the target task and (iii) deep convolutional neural networks.

The healthcare spending in the United States accounts for about 17% of US GDP, making the US the highest healthcare expenditure per capita in the world. There are areas for improvement such as hospital care, which represents the single largest national health expenditure by the type of services. Because hospital operations and manufacturing processes share some similarities at systems level, we believe that some of the theories and techniques developed for manufacturing processes are also applicable to hospital operations. In this work, we examine whether the so-called focused factory theory (i.e., factories that concentrate on narrow range of services or operations produce better products at low costs) is applicable to hospital operations. Specifically, we examine whether the hospitals that are specialized in certain diseases achieve better results in terms of costs and patient outcomes using a large national healthcare cost and utilization project (HCUP) dataset. Challenges in analyzing the HCUP data will be discussed. Pure data-driven machine learning (ML) approaches based on multiple linear regression (MLR), principal component regression (PCR), partial least squares (PLS), Fisher discriminant analysis (FDA), ordinal regression (OR) and logistic regression (LR) will be used to investigate the effects of hospital specialization on hospital performance in terms of cost (measured by total charge) and patient outcome (measured by death of patient during hospitalization). We show that, without domain knowledge, the results from ML approaches are incomplete. The full effects of specialization are revealed only when ML is applied to a model structure that is defined based on domain knowledge. Our results suggest that domain knowledge can play a significant role in machine learning applications and should be incorporated whenever possible.

Electricity demand forecasting is an essential process in the operation and planning procedures of power systems that considerably influences the decisions of utility providers. Main aim of this paper is, first, to examine the relationship between a time series and influential multiple criteria, and, second, to provide long-term electricity demand forecasts in Greece. An original disaggregation or ordinal regression analysis methodological framework is outlined to optimally assess a robust additive value model which is as consistent as possible with a given time series. The accuracy and stability of this modeling approach is guaranteed through the calculation of statistical error measures and robustness analysis indices, respectively. For the case of Greece, the additive value forecasting model is inferred from data related to the training period 1999–2013. The proposed method has been applied for the forecasting of the annual total net electricity demand in the Greek interconnected power system during the following testing period 2014–2016. The model implies that the level of economic growth, represented by the national gross domestic product, imposes the greatest influence on the electricity demand followed by the energy efficiency progress and the weather conditions in the country. The ordinal regression models perform considerably better than the multiple linear (least-squares) regression model, in terms of prediction reliability, resulting into a minimum MAPE equal to 0.74%. The exact method has been also applied for the extraction of electricity demand projections till 2027 based on alternative economic growth scenarios, indicating a constant increase of the electricity demand in Greece.

Head pose estimation has attracted immense research interest recently, as its inherent information significantly improves the performance of face-related applications such as face alignment and face recognition. In this paper, we conduct an in-depth study of head pose estimation and present a multiregression loss function, an $L2$ regression loss combined with an ordinal regression loss, to train a convolutional neural network (CNN) that is dedicated to estimating head poses from RGB images without depth information. The ordinal regression loss is utilized to address the nonstationary property observed as the facial features change with respect to different head pose angles and learn robust features. The $L2$ regression loss leverages these features to provide precise angle predictions for input images. To avoid the ambiguity problem in the commonly used Euler angle representation, we further formulate the head pose estimation problem in quaternions. Our quaternion-based multiregression loss method achieves state-of-the-art performance on the AFLW2000, AFLW test set, and AFW datasets and is closing the gap with methods that utilize depth information on the BIWI dataset.

We integrate sparse radar data into a monocular depth estimation model and introduce a novel preprocessing method for reducing the sparseness and limited field of view provided by radar. We explore the intrinsic error of different radar modalities and show our proposed method results in more data points with reduced error. We further propose a novel method for estimating dense depth maps from monocular 2D images and sparse radar measurements using deep learning based on the deep ordinal regression network by Fu et al. Radar data are integrated by first converting the sparse 2D points to a height-extended 3D measurement and then including it into the network using a late fusion approach. Experiments are conducted on the nuScenes dataset. Our experiments demonstrate state-of-the-art performance in both day and night scenes., Accepted to ICIP2021

Classification and regression techniques are two of the main tasks considered by the Machine Learning area. They mainly depend on the target variable to predict. In this context, ordinal classification represents an intermediate task, which is focused on the prediction of nominal variables where the categories follow a specific intrinsic order given by the problem. Nevertheless, the integration of different algorithms able to solve ordinal classification problems is often unavailable in most of existing Machine Learning software, which hinders the use of new approaches. Therefore, this paper focuses on the incorporation of an ordinal classification algorithm (NSLVOrd) in one of the most complete ordinal regression frameworks, “Ordinal Regression and Classification Algorithms framework (ORCA)” by using both fuzzy rules and the JFML library. The use of NSLVOrd in the ORCA tool as well as a case study with a real database are shown where the obtained results are promising.

The progression of lung cancer implies the intrinsic ordinal relationship of lung nodules at different stages-from benign to unsure then to malignant. This problem can be solved by ordinal regression methods, which is between classification and regression due to its ordinal label. However, existing convolutional neural network (CNN)-based ordinal regression methods only focus on modifying classification head based on a randomly sampled mini-batch of data, ignoring the ordinal relationship resided in the data itself. In this paper, we propose a Meta Ordinal Weighting Network (MOW-Net) to explicitly align each training sample with a meta ordinal set (MOS) containing a few samples from all classes. During the training process, the MOW-Net learns a mapping from samples in MOS to the corresponding class-specific weight. In addition, we further propose a meta cross-entropy (MCE) loss to optimize the network in a meta-learning scheme. The experimental results demonstrate that the MOW-Net achieves better accuracy than the state-of-the-art ordinal regression methods, especially for the unsure class., Accepted by ICASSP 2021

The road extraction task is mainly composed of two subtasks, namely, road detection and road centerline extraction. As the road detection task and road centerline extraction task are strongly correlated, in this paper, we introduce a multitask learning framework to detect roads and extract road centerlines simultaneously. For the road centerline extraction problem, existing works rely either on regression-based methods, or classification-based methods. The regression-based methods suffer from slow convergence and unsatisfactory local solutions. The classification-based methods ignore the fact that the closer the pixel is to the centerline, the higher our tolerance for its misclassification. To overcome these problems, we first convert the road centerline extraction problem into the problem of discrete normalized distance label prediction, which can be resolved by training an ordinal regressor. For the road extraction task, most of the previous studies apply pixel-wise loss function, for example, Cross-Entropy loss, which is not sufficient, as the road has special topology characteristics such as connectivity. Therefore, we propose a road-topology loss function to improve the connectivity and completeness of the extracted road. The road-topology loss function has two key characteristics: (i) The road-topology loss function combines road detection prediction and road centerline extraction prediction to promote the two subtasks to each other by using the correlation between the two subtasks, (ii) The road-topology loss can emphatically penalize gaps that often appear in road detection results and spurious segments that easily appear in centerline extraction results. In this paper, we select the AdamW optimizer to minimize the road-topology loss. Since there is no public dataset, we build a road extraction dataset to evaluate our method. State-of-the-art semantic segmentation networks (LinkNet34, DLinkNet34, DeeplabV3plus) are used as baseline methods to compare with two kinds of method. The first kind of method modifies the baseline method by adding the road centerline extraction task branch based on ordinal regression. The second kind of method uses the road topology loss and has the same network architecture as the first kind of method. For the road detection task, the two kinds of methods improve the baseline methods by up to 3.51% and 11.98% in IoU metric on our test dataset, respectively. For the road centerline extraction task, the two kinds of methods improve the baseline methods by up to 8.22% and 10.9% in the Quality metric on our test dataset.

Coronavirus disease 2019 (COVID-19) has brought huge losses to the world, and it remains a great threat to public health. X-ray computed tomography (CT) plays a central role in the management of COVID-19. Traditional diagnosis with pulmonary CT images is time-consuming and error-prone, which could not meet the need for precise and rapid COVID-19 screening. Nowadays, deep learning (DL) has been successfully applied to CT image analysis, which assists radiologists in workflow scheduling and treatment planning for patients with COVID-19. Traditional methods use cross-entropy as the loss function with a Softmax classifier following a fully-connected layer. Most DL-based classification methods target intraclass relationships in a certain class (early, progressive, severe, or dissipative phases), ignoring the natural order of different phases of the disease progression, i.e., from an early stage and progress to a late stage. To learn both intraclass and interclass relationships among different stages and improve the accuracy of classification, this paper proposes an ensemble learning method based on ordinal regression, which leverages the ordinal information on COVID-19 phases. The proposed method uses multi-binary, neuron stick-breaking (NSB), and soft labels (SL) techniques, and ensembles the ordinal outputs through a median selection. To evaluate our method, we collected 172 confirmed cases. In a 2-fold cross-validation experiment, the accuracy is increased by 22% compared with traditional methods when we use modified ResNet-18 as the backbone. And precision, recall, and F1-score are also improved. The experimental results show that our proposed method achieves a better classification performance than the traditional methods, which helps establish guidelines for the classification of COVID-19 chest CT images.

We present MedMNIST, a collection of 10 pre-processed medical open datasets. MedMNIST is standardized to perform classification tasks on lightweight 28x28 images, which requires no background knowledge. Covering the primary data modalities in medical image analysis, it is diverse on data scale (from 100 to 100,000) and tasks (binary/multi-class, ordinal regression and multi-label). MedMNIST could be used for educational purpose, rapid prototyping, multi-modal machine learning or AutoML in medical image analysis. Moreover, MedMNIST Classification Decathlon is designed to benchmark AutoML algorithms on all 10 datasets; We have compared several baseline methods, including open-source or commercial AutoML tools. The datasets, evaluation code and baseline methods for MedMNIST are publicly available at https://medmnist.github.io/., ISBI 2021. Code and dataset are available at https://medmnist.github.io/

Ordinal regression is aimed at predicting an ordinal class label. In this letter, we consider its semisupervised formulation, in which we have unlabeled data along with ordinal-labeled data to train an ordinal regressor. There are several metrics to evaluate the performance of ordinal regression, such as the mean absolute error, mean zero-one error, and mean squared error. However, the existing studies do not take the evaluation metric into account, restrict model choice, and have no theoretical guarantee. To overcome these problems, we propose a novel generic framework for semisupervised ordinal regression based on the empirical risk minimization principle that is applicable to optimizing all of the metrics mentioned above. In addition, our framework has flexible choices of models, surrogate losses, and optimization algorithms without the common geometric assumption on unlabeled data such as the cluster assumption or manifold assumption. We provide an estimation error bound to show that our risk estimator is consistent. Finally, we conduct experiments to show the usefulness of our framework.

This introduction to visualization techniques and statistical models for second language research focuses on three types of data (continuous, binary, and scalar), helping readers to understand regression models fully and to apply them in their work. Garcia offers advanced coverage of Bayesian analysis, simulated data, exercises, implementable script code, and practical guidance on the latest R software packages. The book, also demonstrating the benefits to the L2 field of this type of statistical work, is a resource for graduate students and researchers in second language acquisition, applied linguistics, and corpus linguistics who are interested in quantitative data analysis.

The rising ubiquity of social media presents a platform for individuals to express suicide ideation, instead of traditional, formal clinical settings. While neural methods for assessing suicide risk on social media have shown promise, a crippling limitation of existing solutions is that they ignore the inherent ordinal nature across fine-grain levels of suicide risk. To this end, we reformulate suicide risk assessment as an Ordinal Regression problem, over the Columbia-Suicide Severity Scale. We propose SISMO, a hierarchical attention model optimized to factor in the graded nature of increasing suicide risk levels, through soft probability distribution since not all wrong risk-levels are equally wrong. We establish the face value of SISMO for preliminary suicide risk assessment on real-world Reddit data annotated by clinical experts. We conclude by discussing the empirical, practical, and ethical considerations pertaining to SISMO in a larger picture, as a human-in-the-loop framework

In digital pathology, nuclei segmentation still remains a challenging task due to the high heterogeneity and variability in the characteristics of nuclei, in particular, the clustered and overlapping nuclei. We propose a distance ordinal regression loss for an improved nuclei instance segmentation in digitized tissue specimen images. A convolutional neural network with two decoder branches is built. The first decoder branch conducts the nuclear pixel prediction and the second branch predicts the distance to the nuclear center, which is utilized to identify the nuclear boundary and to separate out overlapping nuclei. Adopting a distance-decreasing discretization strategy, we recast the problem of the distance prediction as an ordinal regression problem. To evaluate the proposed method, we conduct experiments on multiple independent multitissue histology image datasets. The experimental results on the multi-tissue datasets demonstrate the effectiveness of the proposed model.

One of the most common types of cancer in men is prostate cancer (PCa). Biopsies guided by bi-parametric magnetic resonance imaging (MRI) can aid PCa diagnosis. Previous works have mostly focused on either detection or classification of PCa from MRI. In this work, however, we present a neural network that simultaneously detects and grades cancer tissue in an end-to-end fashion. This is more clinically relevant than the classification goal of the ProstateX-2 challenge. We used the dataset of this challenge for training and testing. We use a 2D U-Net with MRI slices as input and lesion segmentation maps that encode the Gleason Grade Group (GGG), a measure for cancer aggressiveness, as output. We propose a method for encoding the GGG in the model target that takes advantage of the fact that the classes are ordinal. Furthermore, we evaluate methods for incorporating prostate zone segmentations as prior information, and ensembling techniques. The model scored a voxel-wise weighted kappa of $0.446 \pm 0.082$ and a Dice similarity coefficient for segmenting clinically significant cancer of $0.370 \pm 0.046$ , obtained using 5-fold cross-validation. The lesion-wise weighted kappa on the ProstateX-2 challenge test set was $0.13 \pm 0.27$ . We show that our proposed model target outperforms standard multiclass classification and multi-label ordinal regression. Additionally, we present a comparison of methods for further improvement of the model performance.

Diagnosis of liver fibrosis is important for establishing treatment and assessing the risk of carcinogenesis. Ultrasound imaging is an excellent diagnostic method as a screening test in terms of non-invasiveness and simplicity. The purpose of this study was to automatically diagnose liver fibrosis using ultrasound images to reduce the burden on physicians. We proposed and implemented a system for extracting regions of liver parenchyma utilizing U-Net. Using regions of interest, the stage of fibrosis was classified as F0, F1, F2, F3, or F4 utilizing CORALNet, an ordinal regression model based on ResNet18. The effectiveness of the proposed system was verified. The system implemented using U-Net had a maximum mean Dice coefficient of 0.929. The results of classification of liver fibrosis utilizing CORALNet had a mean absolute error (MAE) of 1.22 and root mean square error (RMSE) of 1.60. The per-case results had a MAE of 1.55 and RMSE of 1.34. U-Net extracted regions of liver parenchyma from the images with high accuracy, and CORALNet showed effectiveness using ordinal information to classify fibrosis in the images. As a future task, we will study a model that is less dependent on teaching data.

Regression via classification (RvC) is a common method used for regression problems in deep learning, where the target variable belongs to a set of continuous values. By discretizing the target into a set of non-overlapping classes, it has been shown that training a classifier can improve neural network accuracy compared to using a standard regression approach. However, it is not clear how the set of discrete classes should be chosen and how it affects the overall solution. In this work, we propose that using several discrete data representations simultaneously can improve neural network learning compared to a single representation. Our approach is end-to-end differentiable and can be added as a simple extension to conventional learning methods, such as deep neural networks. We test our method on three challenging tasks and show that our method reduces the prediction error compared to a baseline RvC approach while maintaining a similar model complexity., Comment: Accepted to ICPR2020