Search

We present an algorithm for searching image collections using free-hand sketches that describe the appearance and relative positions of multiple objects. Sketch based image retrieval (SBIR) methods predominantly match queries containing a single, dominant object invariant to its position within an image. Our work exploits drawings as a concise and intuitive representation for specifying entire scene compositions. We train a convolutional neural network (CNN) to encode masked visual features from sketched objects, pooling these into a spatial descriptor encoding the spatial relationships and appearances of objects in the composition. Training the CNN backbone as a Siamese network under triplet loss yields a metric search embedding for measuring compositional similarity which may be efficiently leveraged for visual search by applying product quantization., Comment: ICIP 2021 camera-ready version

Generative adversarial networks (GANs) nowadays are capable of producing images of incredible realism. One concern raised is whether the state-of-the-art GAN's learned distribution still suffers from mode collapse, and what to do if so. Existing diversity tests of samples from GANs are usually conducted qualitatively on a small scale, and/or depends on the access to original training data as well as the trained model parameters. This paper explores to diagnose GAN intra-mode collapse and calibrate that, in a novel black-box setting: no access to training data, nor the trained model parameters, is assumed. The new setting is practically demanded, yet rarely explored and significantly more challenging. As a first stab, we devise a set of statistical tools based on sampling, that can visualize, quantify, and rectify intra-mode collapse. We demonstrate the effectiveness of our proposed diagnosis and calibration techniques, via extensive simulations and experiments, on unconditional GAN image generation (e.g., face and vehicle). Our study reveals that the intra-mode collapse is still a prevailing problem in state-of-the-art GANs and the mode collapse is diagnosable and calibratable in black-box settings. Our codes are available at: https://github.com/VITA-Group/BlackBoxGANCollapse., Comment: This paper has been accepted by Transactions on Multimedia Computing Communications and Applications (TOMM) for publication in 2021

We investigate privacy-preserving, video-based action recognition in deep learning, a problem with growing importance in smart camera applications. A novel adversarial training framework is formulated to learn an anonymization transform for input videos such that the trade-off between target utility task performance and the associated privacy budgets is explicitly optimized on the anonymized videos. Notably, the privacy budget, often defined and measured in task-driven contexts, cannot be reliably indicated using any single model performance because strong protection of privacy should sustain against any malicious model that tries to steal private information. To tackle this problem, we propose two new optimization strategies of model restarting and model ensemble to achieve stronger universal privacy protection against any attacker models. Extensive experiments have been carried out and analyzed. On the other hand, given few public datasets available with both utility and privacy labels, the data-driven (supervised) learning cannot exert its full power on this task. We first discuss an innovative heuristic of cross-dataset training and evaluation, enabling the use of multiple single-task datasets (one with target task labels and the other with privacy labels) in our problem. To further address this dataset challenge, we have constructed a new dataset, termed PA-HMDB51, with both target task labels (action) and selected privacy attributes (skin color, face, gender, nudity, and relationship) annotated on a per-frame basis. This first-of-its-kind video dataset and evaluation protocol can greatly facilitate visual privacy research and open up other opportunities. Our codes, models, and the PA-HMDB51 dataset are available at https://github.com/VITA-Group/PA-HMDB51., Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). arXiv admin note: text overlap with arXiv:1807.08379

In this paper, we show that when Minkowski measure of asymmetry of convex body K of constant width is bigger than α (n-1), K has at least n+1 critical chords, where $$\alpha (n) = \frac{{n + \sqrt {2n(n + 1)} }}{{n + 2}}$$ .

In this paper, we introduce a new measure of asymmetry, called log-Minkowski measure of asymmetry for planar convex bodies in terms of the $$L_0$$ -mixed volume, and show that triangles are the most asymmetric planar convex bodies in the sense of this measure of asymmetry.

In computer vision, superpixels have been widely used as an effective way to reduce the number of image primitives for subsequent processing. But only a few attempts have been made to incorporate them into deep neural networks. One main reason is that the standard convolution operation is defined on regular grids and becomes inefficient when applied to superpixels. Inspired by an initialization strategy commonly adopted by traditional superpixel algorithms, we present a novel method that employs a simple fully convolutional network to predict superpixels on a regular image grid. Experimental results on benchmark datasets show that our method achieves state-of-the-art superpixel segmentation performance while running at about 50fps. Based on the predicted superpixels, we further develop a downsampling/upsampling scheme for deep networks with the goal of generating high-resolution outputs for dense prediction tasks. Specifically, we modify a popular network architecture for stereo matching to simultaneously predict superpixels and disparities. We show that improved disparity estimation accuracy can be obtained on public datasets., 16 pages, 15 figures, to be published in CVPR'20

We introduce a novel principle for self-supervised feature learning based on the discrimination of specific transformations of an image. We argue that the generalization capability of learned features depends on what image neighborhood size is sufficient to discriminate different image transformations: The larger the required neighborhood size and the more global the image statistics that the feature can describe. An accurate description of global image statistics allows to better represent the shape and configuration of objects and their context, which ultimately generalizes better to new tasks such as object classification and detection. This suggests a criterion to choose and design image transformations. Based on this criterion, we introduce a novel image transformation that we call limited context inpainting (LCI). This transformation inpaints an image patch conditioned only on a small rectangular pixel boundary (the limited context). Because of the limited boundary information, the inpainter can learn to match local pixel statistics, but is unlikely to match the global statistics of the image. We claim that the same principle can be used to justify the performance of transformations such as image rotations and warping. Indeed, we demonstrate experimentally that learning to discriminate transformations such as LCI, image warping and rotations, yields features with state of the art generalization capabilities on several datasets such as Pascal VOC, STL-10, CelebA, and ImageNet. Remarkably, our trained features achieve a performance on Places on par with features trained through supervised learning with ImageNet labels., Comment: CVPR 2020 (oral)

Correspondences between frames encode rich information about dynamic content in videos. However, it is challenging to effectively capture and learn those due to their irregular structure and complex dynamics. In this paper, we propose a novel neural network that learns video representations by aggregating information from potential correspondences. This network, named $CPNet$, can learn evolving 2D fields with temporal consistency. In particular, it can effectively learn representations for videos by mixing appearance and long-range motion with an RGB-only input. We provide extensive ablation experiments to validate our model. CPNet shows stronger performance than existing methods on Kinetics and achieves the state-of-the-art performance on Something-Something and Jester. We provide analysis towards the behavior of our model and show its robustness to errors in proposals., CVPR 2019 (Oral)

In this paper, we consider a measure of asymmetry for Reuleaux polygons, and show that the n-th (\(n \ge 3, n \;\text {odd}\)) regular Reuleaux polygons are the most symmetric ones among all n-th Reuleaux polygons. As a byproduct, we show that the Reuleaux triangles are the most asymmetric planar convex bodies of constant width.

Font selection is one of the most important steps in a design workflow. Traditional methods rely on ordered lists which require significant domain knowledge and are often difficult to use even for trained professionals. In this paper, we address the problem of large-scale tag-based font retrieval which aims to bring semantics to the font selection process and enable people without expert knowledge to use fonts effectively. We collect a large-scale font tagging dataset of high-quality professional fonts. The dataset contains nearly 20,000 fonts, 2,000 tags, and hundreds of thousands of font-tag relations. We propose a novel generative feature learning algorithm that leverages the unique characteristics of fonts. The key idea is that font images are synthetic and can therefore be controlled by the learning algorithm. We design an integrated rendering and learning process so that the visual feature from one image can be used to reconstruct another image with different text. The resulting feature captures important font design details while is robust to nuisance factors such as text. We propose a novel attention mechanism to re-weight the visual feature for joint visual-text modeling. We combine the feature and the attention mechanism in a novel recognition-retrieval model. Experimental results show that our method significantly outperforms the state-of-the-art for the important problem of large-scale tag-based font retrieval., Comment: accepted by ICCV 2019

LiveSketch is a novel algorithm for searching large image collections using hand-sketched queries. LiveSketch tackles the inherent ambiguity of sketch search by creating visual suggestions that augment the query as it is drawn, making query specification an iterative rather than one-shot process that helps disambiguate users' search intent. Our technical contributions are: a triplet convnet architecture that incorporates an RNN based variational autoencoder to search for images using vector (stroke-based) queries; real-time clustering to identify likely search intents (and so, targets within the search embedding); and the use of backpropagation from those targets to perturb the input stroke sequence, so suggesting alterations to the query in order to guide the search. We show improvements in accuracy and time-to-task over contemporary baselines using a 67M image corpus., Comment: Accepted to CVPR 2019

We address the problem of image feature learning for scene text recognition. The image features in the state-of-the-art methods are learned from large-scale synthetic image datasets. However, most meth- ods only rely on outputs of the synthetic data generation process, namely realistically looking images, and completely ignore the rest of the process. We propose to leverage the parameters that lead to the output images to improve image feature learning. Specifically, for every image out of the data generation process, we obtain the associated parameters and render another “clean” image that is free of select distortion factors that are ap- plied to the output image. Because of the absence of distortion factors, the clean image tends to be easier to recognize than the original image which can serve as supervision. We design a multi-task network with an encoder-discriminator-generator architecture to guide the feature of the original image toward that of the clean image. The experiments show that our method significantly outperforms the state-of-the-art methods on standard scene text recognition benchmarks in the lexicon-free cate- gory. Furthermore, we show that without explicit handling, our method works on challenging cases where input images contain severe geometric distortion, such as text on a curved path.

Understanding the difference between malignant and benign pancreatic masses is critical in terms of diagnosis, although this is difficult to determine in clinical practice. The contrast-enhanced harmonic endoscopic ultrasound (CH-EUS) technique was introduced in 2010, although, to the best of the authors' knowledge, there has been no systematic review or meta-analysis to date evaluating its diagnostic performance for the differentiation of pancreatic masses. The aim of the present study was to systematically evaluate the diagnostic performance of CH-EUS for the differentiation of pancreatic masses. Search key words and inclusion and exclusion criteria were initially presented. Two independent authors read and extracted the relevant information from the included studies. Disagreements were resolved through discussion with another two experienced authors. Metadisc and Stata software were used for the meta-analysis and the evaluation of bias. A total of 16 studies comprising 1,325 patients were included in this meta-analysis. The pooled sensitivity, specificity, positive likelihood ratio, negative likelihood ratio and diagnostic odds ratio of CH-EUS were used to distinguish between malignant and benign tumors, and the values obtained were 93% [95% confidence interval (CI): 91-94%], 84% (95% CI: 80-87%), 5.58 (95% CI: 3.90-7.97), 0.09 (95% CI: 0.07-0.11) and 72.56 (95% CI: 48.93-107.60), respectively. The area under the summary receiver operating characteristic curve was determined to be 0.96. No publication bias was identified in this meta-analysis. Taken together, these results confirm that CH-EUS has a high accuracy rate for distinguishing between benign and malignant pancreatic space-occupying lesions, and it may therefore be used as an effective diagnostic tool for pancreatic masses.

Typography is a fundamental graphic design component and also a ubiquitous art form that affects our understanding and perception of what we read. Thousands of different font faces have been created with enormous variations in the characters. Graphic designers have the desire to identify the fonts they encounter in daily life for later use. While they might take a photo of the text of a particularly interesting font and seek out an expert to identify the font, the manual identification process is extremely tedious and error prone. Several websites allow users to search and recognize fonts by font similarity, including Identifont, MyFonts, WhatTheFont, and Fontspring. All of them rely on tedious humans interactions and high-quality manual preprocessing of images, and the accuracies are still unsatisfactory. On the other hand, the majority of font selection interfaces in existing software applications are simple linear lists, while exhaustively exploring the entire space of fonts using an alphabetical listing is unrealistic for most users.

In this paper, the authors define a harmonic Orlicz combination and a dual Orlicz mixed volume of star bodies, and then establish the dual Orlicz-Minkowski mixedvolume inequality and the dual Orlicz-Brunn-Minkowksi inequality.

In this paper, we study the Minkowski measure of asymmetry for n-dimensional convex bodies of revolution ( n⩾3 ). We show that among all n-dimensional convex bodies of revolution, the bodies which generated by isosceles triangles are the most asymmetric ones. Also, we study the asymmetry for n-dimensional constant width bodies of revolution.

This paper introduces the problem of automatic font pairing . Font pairing is an important design task that is difficult for novices. Given a font selection for one part of a document (e.g., header), our goal is to recommend a font to be used in another part (e.g., body) such that the two fonts used together look visually pleasing. There are three main challenges in font pairing. First, this is a fine-grained problem, in which the subtle distinctions between fonts may be important. Second, rules and conventions of font pairing given by human experts are difficult to formalize. Third, font pairing is an asymmetric problem in that the roles played by header and body fonts are not interchangeable. To address these challenges, we propose automatic font pairing through learning visual relationships from large-scale human-generated font pairs. We introduce a new database for font pairing constructed from millions of PDF documents available on the Internet. We propose two font pairing algorithms: dual-space $k$ -NN and asymmetric similarity metric learning (ASML). These two methods automatically learn fine-grained relationships from large-scale data. We also investigate several baseline methods based on the rules from professional designers. Experiments and user studies demonstrate the effectiveness of our proposed dataset and methods.

In 1996, E. Lutwak extended the important concept of geominimal surface area to Lp version, which serves as a bridge connecting a number of areas of geometry: affine differential geometry, relative differential geometry, and Minkowskian geometry. In this paper, by using the concept of Orlicz mixed volume, we extend geominimal surface area to the Orlicz version and give some properties and an isoperimetric inequalities for the Orlicz geominimal surface areas. Mathematics subject classification (2010): 52A39, 52A40.

We study the skip-thought model with neighborhood information as weak supervision. More specifically, we propose a skip-thought neighbor model to consider the adjacent sentences as a neighborhood. We train our skip-thought neighbor model on a large corpus with continuous sentences, and then evaluate the trained model on 7 tasks, which include semantic relatedness, paraphrase detection, and classification benchmarks. Both quantitative comparison and qualitative investigation are conducted. We empirically show that, our skip-thought neighbor model performs as well as the skip-thought model on evaluation tasks. In addition, we found that, incorporating an autoencoder path in our model didn’t aid our model to perform better, while it hurts the performance of the skip-thought model.

In this paper, we prove a sharp Rogers–Shephard type inequality for the Orlicz-difference body of planar convex bodies, which extend the works of Bianchini and Colesanti (Proc. Amer. Math. Soc. 138(7) (2008) 2575–2582).

The Orlicz Brunn–Minkowski theory originated with the work of Lutwak, Yang, and Zhang in 2010. In this paper, we first introduce the Orlicz addition of convex bodies containing the origin in their interiors, and then extend the L p Brunn–Minkowski inequality to the Orlicz Brunn–Minkowski inequality. Furthermore, we extend the L p Minkowski mixed volume inequality to the Orlicz mixed volume inequality by using the Orlicz Brunn–Minkowski inequality.

The $$L_p$$ metrics, extensions of the Hausdorff metric for convex bodies, were investigated by Vitale in 1985. In this paper, we extend $$L_p$$ metrics to Orlicz metrics, and show that these Orlicz metrics generate the same topology as the Hausdorff one in the space of all convex bodies (i.e., non-empty compact convex subsets), consequently, the space of all convex bodies with the Orlicz metric is a complete, separable metric space. Furthermore, we also show that the space of all convex bodies with the Orlicz metric is a metric segment space.

In this paper we study the $$1$$ -measure of asymmetry for convex bodies of constant width.

In 1988, H. Groemer gave a stability theorem for the area of convex domains of constant width. In this paper, we obtain a stability theorem for the well-known Minkowski measure of asymmetry for convex domains of constant width.

Visual-semantic embedding models have been recently proposed and shown to be effective for image classification and zero-shot learning, by mapping images into a continuous semantic label space. Although several approaches have been proposed for single-label embedding tasks, handling images with multiple labels (which is a more general setting) still remains an open problem, mainly due to the complex underlying corresponding relationship between image and its labels. In this work, we present Multi-Instance visual-semantic Embedding model (MIE) for embedding images associated with either single or multiple labels. Our model discovers and maps semantically-meaningful image subregions to their corresponding labels. And we demonstrate the superiority of our method over the state-of-the-art on two tasks, including multi-label image annotation and zero-shot learning.

We propose an algorithm to perform causal inference of the state of a dynamical model when the measurements are corrupted by outliers. While the optimal (maximum-likelihood) solution has doubly exponential complexity due to the combinatorial explosion of possible choices of inliers, we exploit the structure of the problem to design a sampling-based algorithm that has constant complexity. We derive our algorithm from the equations of the optimal filter, which makes our approximation explicit. Our work is motivated by real-time tracking and the estimation of structure from motion (SFM). We test our algorithm for on-line outlier rejection both for tracking and for SFM. We show that our approach can tolerate a large proportion of outliers, whereas previous causal robust statistical inference methods failed with less than half as many. Our work can be thought of as the extension of random sample consensus algorithms to dynamic data, or as the implementation of pseudo-Bayesian filtering algorithms in a sampling framework. © 2005 IEEE.

“Interactive Inverse 3D Modeling” is a user-guided approach to shape construction and redesign that extracts well-structured, parameterized, procedural descriptions from unstructured, hierarchically flat input data, such as point clouds, boundary representation meshes, or even multiple pictorial views of a given inspirational prototype. This approach combines traditional “forward” 3D modeling tools with a system of user-guided extraction modules and optimization routines. With a few cursor strokes users can express their preferences of the type of modeling primitives to be used in a particular area of the given prototype to be approximated, and they can also select the degree of parameterization associated with each modeling routine. The results are then pliable, structured descriptions that are well suited to implement the particular design modifications intended by the user.

In a previous paper, we showed that for regular Reuleaux polygons Rn the equality \({{\rm as}_\infty(R_n) = 1/(2\cos \frac\pi{2n} -1)}\) holds, where \({{\rm as}_\infty(\cdot)}\) denotes the Minkowski measure of asymmetry for convex bodies, and \({{\rm as}_\infty(K)\leq \frac 12(\sqrt{3}+1)}\) for all convex domains K of constant width, with equality holds iff K is a Reuleaux triangle. In this paper, we investigate the Minkowski measures of asymmetry among all Reuleaux polygons of order n and show that regular Reuleaux polygons of order n (n ≥ 3 and odd) have the minimal Minkowski measure of asymmetry.

In a previous paper, we showed that for all convex bodies K of constant width in \({\mathbb{R}^n, 1 \leq {\rm as}_\infty(K) \leq \frac{n+\sqrt{2n(n+1)}}{n+2}}\) , where as∞(·) denotes the Minkowski measure of asymmetry, with the equality holding on the right-hand side if K is a completion of a regular simplex, and asked whether or not the completions of regular simplices are the only bodies for the equality. A positive answer is given in this short note.

Image feature representation plays an essential role in image recognition and related tasks. The current state-of-the-art feature learning paradigm is supervised learning from labeled data. However, this paradigm requires large-scale category labels, which limits its applicability to domains where labels are hard to obtain. In this paper, we propose a new data-driven feature learning paradigm which does not rely on category labels. Instead, we learn from user behavior data collected on social media. Concretely, we use the image relationship discovered in the latent space from the user behavior data to guide the image feature learning. We collect a large-scale image and user behavior dataset from Behance.net. The dataset consists of 1.9 million images and over 300 million view records from 1.9 million users. We validate our feature learning paradigm on this dataset and find that the learned feature significantly outperforms the state-of-the-art image features in learning better image similarities. We also show that the learned feature performs competitively on various recognition benchmarks.

As font is one of the core design concepts, automatic font identification and similar font suggestion from an image or photo has been on the wish list of many designers. We study the Visual Font Recognition (VFR) problem, and advance the state-of-the-art remarkably by developing the DeepFont system. First of all, we build up the first available large-scale VFR dataset, named AdobeVFR, consisting of both labeled synthetic data and partially labeled real-world data. Next, to combat the domain mismatch between available training and testing data, we introduce a Convolutional Neural Network (CNN) decomposition approach, using a domain adaptation technique based on a Stacked Convolutional Auto-Encoder (SCAE) that exploits a large corpus of unlabeled real-world text images combined with synthetic data preprocessed in a specific way. Moreover, we study a novel learning-based model compression approach, in order to reduce the DeepFont model size without sacrificing its performance. The DeepFont system achieves an accuracy of higher than 80% (top-5) on our collected dataset, and also produces a good font similarity measure for font selection and suggestion. We also achieve around 6 times compression of the model without any visible loss of recognition accuracy., Comment: To Appear in ACM Multimedia as a full paper

In this paper, we study the so-called mean Minkowski measures, proposed and studied by Toth in a series of papers, for convex bodies of constant width. We show that, with respect to the mean Minkowski measure, the completions of regular simplices are, as well as for many other measures, the most asymmetric ones among all convex bodies of constant width.

Light Emitting Diodes (LEDs) are increasingly being designed into lighting products for general illumination and are now gaining traction in the market. Several LED products are already available, but have generally been accepted only for low light level applications because of the low lumen output and poor quality of light for many of the products. Much of the reason for the low lumen output is the limited cooling capacity of natural convection which is most commonly used. Unlike incandescent, halogen based, or other traditional lighting technologies, LEDs need proper thermal management to increase the lumen output, maintain quality of light and ensure high reliability and useful life. This paper investigates synthetic jet technology as a viable option for forced convection cooling as an alternative for natural convection in LED products to enable higher lumen designs without compromising form factor, quality of light and reliability.

We begin by presenting an active contour model which utilizes the Mumford-Shah energy functional for the purpose of piecewise smooth image segmentation. We then show how the use of simultaneous piecewise smooth image segmentation on a set of calibrated 2D images of a common 3D scene may be utilized for reconstructing the unknown shapes and radiances of scene objects. To do so, we must lift the the support of the unknown smooth functions in the traditional Mumford-Shah framework, which will now represent the unknown radiance of scene objects in this application, onto a manifold which will represent the unknown shape of scene objects. This constitutes a significant mathematical departure from the traditional Mumford-Shah model since the unknown functions now live directly on the unknown geometric surfaces rather than their surrounding ambient space. The intution, however, follows from the original model in that the estimates must closely match the observed data while maintaining a high degree of smoothness both in the estimated functions as well as the estimated geometry.

We develop an efficient algorithm to track point features supported by image patches undergoing affine deformations and changes in illumination. The algorithm is based on a combined model of geometry and photometry, that is used to track features as well as to detect outliers in a hypothesis testing framework. The algorithm runs in real time on a personal computer; and is available to the public.

We present a system to insert virtual objects into real image sequences in real time. The system consists of offthe- shelf hardware (a camera connected to a Pentium PC) and software to (a) automatically select and track region features despite changes in illumination, (b) estimate threedimensional position and orientation of surface patches relative to an inertial reference frame despite individual pointfeatures appearing and disappearing, (c) insert a texturemapped virtual object into the scene so as to make it appear to be part of the scene and moving with it. This is all done in real time. The multi-thread C++ code, which is readily interfaced with a frame grabber as well as Matlab for development, will be made available to the public at the demonstration.