Search

We consider sequential or active ranking of a set of n items based on noisy pairwise comparisons. Items are ranked according to the probability that a given item beats a randomly chosen item, and ranking refers to partitioning the items into sets of pre-specified sizes according to their scores. This notion of ranking includes as special cases the identification of the top-k items and the total ordering of the items. We first analyze a sequential ranking algorithm that counts the number of comparisons won, and uses these counts to decide whether to stop, or to compare another pair of items, chosen based on confidence intervals specified by the data collected up to that point. We prove that this algorithm succeeds in recovering the ranking using a number of comparisons that is optimal up to logarithmic factors. This guarantee does not require any structural properties of the underlying pairwise probability matrix, unlike a significant body of past work on pairwise ranking based on parametric models such as the Thurstone or Bradley-Terry-Luce models. It has been a long-standing open question as to whether or not imposing these parametric assumptions allows for improved ranking algorithms. For stochastic comparison models, in which the pairwise probabilities are bounded away from zero, our second contribution is to resolve this issue by proving a lower bound for parametric models. This shows, perhaps surprisingly, that these popular parametric modeling choices offer at most logarithmic gains for stochastic comparisons., Comment: improved log factor in main result; added discussion on comparison probabilities close to zero; added numerical results

We consider data in the form of pairwise comparisons of n items, with the goal of precisely identifying the top k items for some value of k < n, or alternatively, recovering a ranking of all the items. We analyze the Copeland counting algorithm that ranks the items in order of the number of pairwise comparisons won, and show it has three attractive features: (a) its computational efficiency leads to speed-ups of several orders of magnitude in computation time as compared to prior work; (b) it is robust in that theoretical guarantees impose no conditions on the underlying matrix of pairwise-comparison probabilities, in contrast to some prior work that applies only to the BTL parametric model; and (c) it is an optimal method up to constant factors, meaning that it achieves the information-theoretic limits for recovering the top k-subset. We extend our results to obtain sharp guarantees for approximate recovery under the Hamming distortion metric, and more generally, to any arbitrary error requirement that satisfies a simple and natural monotonicity condition., Changes in version 2: In addition to recovery in the exact and Hamming metrics, v2 analyzes a general, abstract recovery criterion based on a notion of "allowed sets"

Chronic nonhealing skin wounds often contain bacterial biofilms that prevent normal wound healing and closure and present challenges to the use of conventional wound dressings. We investigated inhibition of Pseudomonas aeruginosa biofilm formation, a common pathogen of chronic skin wounds, on a commercially available biological wound dressing. Building on prior reports, we examined whether the amino acid tryptophan would inhibit P. aeruginosa biofilm formation on the three-dimensional surface of the biological dressing. Bacterial biomass and biofilm polysaccharides were quantified using crystal violet staining or an enzyme linked lectin, respectively. Bacterial cells and biofilm matrix adherent to the wound dressing were visualized through scanning electron microscopy. D-/L-tryptophan inhibited P. aeruginosa biofilm formation on the wound dressing in a dose dependent manner and was not directly cytotoxic to immortalized human keratinocytes although there was some reduction in cellular metabolism or enzymatic activity. More importantly, D-/L-tryptophan did not impair wound healing in a splinted skin wound murine model. Furthermore, wound closure was improved when D-/L-tryptophan treated wound dressing with P. aeruginosa biofilms were compared with untreated dressings. These findings indicate that tryptophan may prove useful for integration into wound dressings to inhibit biofilm formation and promote wound healing.

Detonation-based combustors leverage the higher thermodynamic efficiency of the Atkinson cycle compared to the traditional deflagration-based combustion of the Brayton cycle. The rotating detonation engine (RDE) has one or more shock waves rotating around an annulus. The RDE can theoretically be 20% more thermally efficient than a traditional deflagration-based cycle. A RDE was modeled in Numerical Propulsion System Simulation (NPSS) based on a model developed in Microsoft Excel. The thermodynamic analysis of the RDE in these models is broken into four streams. Empirical models were used to find the percentage of the total flow in each stream. The pre-detonation pressure was iterated until the entrance mass flow calculations matched the exit mass flow calculations. A parametric analysis was used to compare the variation in specific impulse from the NPSS model to the Microsoft Excel model and other published results. The RDE has a peak air-breathing engine specific impulse of approximately 5,500 sec and a peak rocket engine specific impulse of approximately 150 sec.

The growing need for labeled training data has made crowdsourcing an important part of machine learning. The quality of crowdsourced labels is, however, adversely affected by three factors: (1) the workers are not experts; (2) the incentives of the workers are not aligned with those of the requesters; and (3) the interface does not allow workers to convey their knowledge accurately, by forcing them to make a single choice among a set of options. In this paper, we address these issues by introducing approval voting to utilize the expertise of workers who have partial knowledge of the true answer, and coupling it with a ("strictly proper") incentive-compatible compensation mechanism. We show rigorous theoretical guarantees of optimality of our mechanism together with a simple axiomatic characterization. We also conduct preliminary empirical studies on Amazon Mechanical Turk which validate our approach.

There is a rapidly increasing interest in crowdsourcing for data labeling. By crowdsourcing, a large number of labels can be often quickly gathered at low cost. However, the labels provided by the crowdsourcing workers are usually not of high quality. In this paper, we propose a minimax conditional entropy principle to infer ground truth from noisy crowdsourced labels. Under this principle, we derive a unique probabilistic labeling model jointly parameterized by worker ability and item difficulty. We also propose an objective measurement principle, and show that our method is the only method which satisfies this objective measurement principle. We validate our method through a variety of real crowdsourcing datasets with binary, multiclass or ordinal labels., Comment: 31 pages

Hepatocellular carcinoma (HCC) usually occurs in patients with underlying risk factors such as liver cirrhosis and chronic hepatitis B. Although patients with Crohn’s disease (CD) are at an increased risk to develop malignancies such as colon cancer, the incidence of HCC in this population is extremely rare. We report a case of 62-year-old male with long history of CD treated with azathioprine (AZA) and aminosalicylic acid (ASA) who was incidentally diagnosed with HCC, for which left hepatectomy was done. Four years later during routine follow-up, patient had another hepatic lesion and underwent resection of the mass. The mechanism of occurrence of HCC in patient with CD is still controversial and may include immune mediated changes and medication related complications. AZA was reported in all case reports of CD that developed HCC. Through this report we hope to explore the complex pathophysiological mechanisms contributing to the development of HCC in the Crohn’s disease patient population.

Human computation or crowdsourcing involves joint inference of the ground-truth-answers and the worker-abilities by optimizing an objective function, for instance, by maximizing the data likelihood based on an assumed underlying model. A variety of methods have been proposed in the literature to address this inference problem. As far as we know, none of the objective functions in existing methods is convex. In machine learning and applied statistics, a convex function such as the objective function of support vector machines (SVMs) is generally preferred, since it can leverage the high-performance algorithms and rigorous guarantees established in the extensive literature on convex optimization. One may thus wonder if there exists a meaningful convex objective function for the inference problem in human computation. In this paper, we investigate this convexity issue for human computation. We take an axiomatic approach by formulating a set of axioms that impose two mild and natural assumptions on the objective function for the inference. Under these axioms, we show that it is unfortunately impossible to ensure convexity of the inference problem. On the other hand, we show that interestingly, in the absence of a requirement to model "spammers", one can construct reasonable objective functions for crowdsourcing that guarantee convex inference., AAAI 2015

When eliciting judgements from humans for an unknown quantity, one often has the choice of making direct-scoring (cardinal) or comparative (ordinal) measurements. In this paper we study the relative merits of either choice, providing empirical and theoretical guidelines for the selection of a measurement scheme. We provide empirical evidence based on experiments on Amazon Mechanical Turk that in a variety of tasks, (pairwise-comparative) ordinal measurements have lower per sample noise and are typically faster to elicit than cardinal ones. Ordinal measurements however typically provide less information. We then consider the popular Thurstone and Bradley-Terry-Luce (BTL) models for ordinal measurements and characterize the minimax error rates for estimating the unknown quantity. We compare these minimax error rates to those under cardinal measurement models and quantify for what noise levels ordinal measurements are better. Finally, we revisit the data collected from our experiments and show that fitting these models confirms this prediction: for tasks where the noise in ordinal measurements is sufficiently low, the ordinal approach results in smaller errors in the estimation.

Crowdsourcing has gained immense popularity in machine learning applications for obtaining large amounts of labeled data. Crowdsourcing is cheap and fast, but suffers from the problem of low-quality data. To address this fundamental challenge in crowdsourcing, we propose a simple payment mechanism to incentivize workers to answer only the questions that they are sure of and skip the rest. We show that surprisingly, under a mild and natural "no-free-lunch" requirement, this mechanism is the one and only incentive-compatible payment mechanism possible. We also show that among all possible incentive-compatible mechanisms (that may or may not satisfy no-free-lunch), our mechanism makes the smallest possible payment to spammers. We further extend our results to a more general setting in which workers are required to provide a quantized confidence for each question. Interestingly, this unique mechanism takes a "multiplicative" form. The simplicity of the mechanism is an added benefit. In preliminary experiments involving over 900 worker-task pairs, we observe a significant drop in the error rates under this unique mechanism for the same or lower monetary expenditure.