Search

Cities undergo constant change and their industrial spaces are particularly impacted on by new technologies, changing industries and the vagaries of the global and national economies. This paper reports on an investigation of the changing commercial and industrial landscape in Cape Town, South Africa, by focussing on commercial park spaces, specifically those that have been purposely named as business parks, industrial parks and office parks. The study is grounded in the literature on the changing spatial economy of Cape Town. A comprehensive database of commercial parks was created from a host of online sources and documents as well as a field survey. The spatial distribution of commercial parks was analysed in relation to the 11 districts mapped by a previous spatial economic study of the city. It was found that while the commercial parks were predominantly located in four districts, a nuanced locational analysis of the three different types of parks provided evidence of subtle differences in their distribution. The commercial parks are located both in traditional industrial suburbs and in newer commercial spaces and, although most are new developments, some are housed in old industrial spaces that have been converted. Almost half of the commercial parks are managed as sectional title schemes which mirror the management model of gated communities. Purposely named commercial parks are essentially a post-apartheid phenomenon as the majority are post-1994 developments.

A decade of unprecedented progress in artificial intelligence (AI) has demonstrated the potential for many fields—including medicine—to benefit from the insights that AI techniques can extract from data. Here we survey recent progress in the development of modern computer vision techniques—powered by deep learning—for medical applications, focusing on medical imaging, medical video, and clinical deployment. We start by briefly summarizing a decade of progress in convolutional neural networks, including the vision tasks they enable, in the context of healthcare. Next, we discuss several example medical imaging applications that stand to benefit—including cardiology, pathology, dermatology, ophthalmology–and propose new avenues for continued work. We then expand into general medical video, highlighting ways in which clinical workflows can integrate computer vision to enhance care. Finally, we discuss the challenges and hurdles required for real-world clinical deployment of these technologies.

For newly diagnosed breast cancer, estrogen receptor status (ERS) is a key molecular marker used for prognosis and treatment decisions. During clinical management, ERS is determined by pathologists from immunohistochemistry (IHC) staining of biopsied tissue for the targeted receptor, which highlights the presence of cellular surface antigens. This is an expensive, time-consuming process which introduces discordance in results due to variability in IHC preparation and pathologist subjectivity. In contrast, hematoxylin and eosin (H&E) staining—which highlights cellular morphology—is quick, less expensive, and less variable in preparation. Here we show that machine learning can determine molecular marker status, as assessed by hormone receptors, directly from cellular morphology. We develop a multiple instance learning-based deep neural network that determines ERS from H&E-stained whole slide images (WSI). Our algorithm—trained strictly with WSI-level annotations—is accurate on a varied, multi-country dataset of 3,474 patients, achieving an area under the curve (AUC) of 0.92 for sensitivity and specificity. Our approach has the potential to augment clinicians’ capabilities in cancer prognosis and theragnosis by harnessing biological signals imperceptible to the human eye., Determination of estrogen receptor status (ERS) in breast cancer tissue requires immunohistochemistry, which is sensitive to the vagaries of sample processing and the subjectivity of pathologists. Here the authors present a deep learning model that determines ERS from H&E stained tissue, which could improve oncology decisions in under-resourced settings.

Bypassing nature’s evolutionary trajectory,de novoprotein generation—defined as creating artificial protein sequences from scratch—could enable breakthrough solutions for biomedical and environmental challenges. Viewing amino acid sequences as a language, we demonstrate that a deep learning-based language model can generate functional artificial protein sequences across families, akin to generating grammatically and semantically correct natural language sentences on diverse topics. Our protein language model is trained by simply learning to predict the next amino acid for over 280 million protein sequences from thousands of protein families, without biophysical or coevolutionary modeling. We experimentally evaluate model-generated artificial proteins on five distinct antibacterial lysozyme families. Artificial proteins show similar activities and catalytic efficiencies as representative natural lysozymes, including hen egg white lysozyme, while reaching as low as 44% identity to any known naturally-evolved protein. The X-ray crystal structure of an enzymatically active artificial protein recapitulates the conserved fold and positioning of active site residues found in natural proteins. We demonstrate our language model’s ability to be adapted to different protein families by accurately predicting the functionality of artificial chorismate mutase and malate dehydrogenase proteins. These results indicate that neural language models successfully performde novoprotein generation across protein families and may prove to be a tool to shortcut evolution.

Recent advances in self-supervised learning (SSL) have largely closed the gap with supervised ImageNet pretraining. Despite their success these methods have been primarily applied to unlabeled ImageNet images, and show marginal gains when trained on larger sets of uncurated images. We hypothesize that current SSL methods perform best on iconic images, and struggle on complex scene images with many objects. Analyzing contrastive SSL methods shows that they have poor visual grounding and receive poor supervisory signal when trained on scene images. We propose Contrastive Attention-Supervised Tuning (CAST) to overcome these limitations. CAST uses unsupervised saliency maps to intelligently sample crops, and to provide grounding supervision via a Grad-CAM attention loss. Experiments on COCO show that CAST significantly improves the features learned by SSL methods on scene images, and further experiments show that CAST-trained models are more robust to changes in backgrounds. Our code is available at https://github.com/salesforce/CAST/.

The proliferation of artificial intelligence based systems in all walks of life raises concerns about their safety and robustness, especially for cyber-physical systems including multiple machine learning components. In this paper, we introduce robustness contracts as a framework for compositional specification and reasoning about the robustness of cyber-physical systems based on neural network (NN) components. Robustness contracts can encompass and generalize a variety of notions of robustness which were previously proposed in the literature. They can seamlessly apply to NN-based perception as well as deep reinforcement learning (RL)-enabled control applications. We present a sound and complete algorithm that can efficiently verify the satisfaction of a class of robustness contracts on NNs by leveraging notions from Lagrangian duality to identify system configurations that violate the contracts. We illustrate the effectiveness of our approach on the verification of NN-based perception systems and deep RL-based control systems.

Generative modeling for protein engineering is key to solving fundamental problems in synthetic biology, medicine, and material science. We pose protein engineering as an unsupervised sequence generation problem in order to leverage the exponentially growing set of proteins that lack costly, structural annotations. We train a 1.2B-parameter language model, ProGen, on ∼280M protein sequences conditioned on taxonomic and keyword tags such as molecular function and cellular component. This provides ProGen with an unprecedented range of evolutionary sequence diversity and allows it to generate with fine-grained control as demonstrated by metrics based on primary sequence similarity, secondary structure accuracy, and conformational energy.

Traditional tools to map the distribution of urban green space have been hindered by either high cost and labour inputs or poor spatial resolution given the complex spatial structure of urban landscapes. What’s more, those tools do not observe the urban landscape from a perspective in which citizens experience a city. We test a novel application of computer vision to quantify urban tree cover at the street-level. We do so by utilizing the open-source image data of city streetscapes that is now abundant (Google Street View). We show that a multi-step computer vision algorithm segments and quantifies the percent of tree cover in streetscape images to a high degree of precision. By then modelling the relationship between neighbouring images along city street segments, we are able to extend this image representation and estimate the amount of perceived tree cover in city streetscapes to a relatively high level of accuracy for an entire city. Though not a replacement for high resolution remote sensing (e.g., aerial LiDAR) or intensive field surveys, the method provides a new multi-feature metric of urban tree cover that quantifies tree presence and distribution from the same viewpoint in which citizens experience and see the urban landscape.

Which neighborhoods experience physical improvements? In this paper, we introduce a computer vision method to measure changes in the physical appearances of neighborhoods from time-series street-level imagery. We connect changes in the physical appearance of five US cities with economic and demographic data and find three factors that predict neighborhood improvement. First, neighborhoods that are densely populated by college-educated adults are more likely to experience physical improvements-an observation that is compatible with the economic literature linking human capital and local success. Second, neighborhoods with better initial appearances experience, on average, larger positive improvements-an observation that is consistent with "tipping" theories of urban change. Third, neighborhood improvement correlates positively with physical proximity to the central business district and to other physically attractive neighborhoods-an observation that is consistent with the "invasion" theories of urban sociology. Together, our results provide support for three classical theories of urban change and illustrate the value of using computer vision methods and street-level imagery to understand the physical dynamics of cities.

Imaging in radio astronomy is usually carried out with a single-dish radio telescope doing a raster scan of a region of the sky or with an interferometer that samples the visibility function of the sky brightness. Mosaic observations are the current standard for imaging large fields of view with an interferometer and multi-frequency observations are now routinely carried out with both types of telescopes to increase the continuum imaging sensitivity and to probe spectral structure. This paper describes an algorithm to combine wideband data from these two types of telescopes in a joint iterative reconstruction scheme that can be applied to spectral cube or wideband multi-term imaging both for narrow fields of view as well as mosaics. Our results demonstrate the ability to prevent instabilities and error that typically arise when wide-band or joint mosaicing algorithms are presented with spatial and spectral structure that is inadequetely sampled by the interferometer alone. For comparable noise levels in the single dish and interferometer data, the numerical behaviour of this algorithm is expected to be similar to the idea of generating artificial visibilities from single dish data. However, our discussed implementation is simpler and more flexible in terms of applying relative data weighting schemes to match noise levels while preserving flux accuracy, fits within standard iterative image reconstruction frameworks, is fully compatible with wide-field and joint mosaicing gridding algorithms that apply corrections specific to the interferometer data and may be configured to enable spectral cube and wideband multi-term deconvolution for single-dish data alone., Comment: 15 pages, 19 figures. Accepted to The Astronomical Journal (16 April 2019)

New, “big data” sources allow measurement of city characteristics and outcome variables at higher collection frequencies and more granular geographic scales than ever before. However, big data will not solve large urban social science questions on its own. Big urban data has the most value for the study of cities when it allows measurement of the previously opaque, or when it can be coupled with exogenous shocks to people or place. We describe a number of new urban data sources and illustrate how they can be used to improve the study and function of cities. We first show how Google Street View images can be used to predict income in New York City, suggesting that similar imagery data can be used to map wealth and poverty in previously unmeasured areas of the developing world. We then discuss how survey techniques can be improved to better measure willingness to pay for urban amenities. Finally, we explain how Internet data is being used to improve the quality of city services. (JEL R1, C8, C18)

How much does the appearance of a house, or its neighbors, impact its price? Do events that impact the incentives facing homeowners, like foreclosure, impact the maintenance and appearance of a home? Using computer vision techniques, we find that a one standard deviation improvement in the appearance of a home in Boston is associated with a .16 log point increase in the home’s value, or about $68,000 at the sample mean. The additional predictive power created by images is small relative to location and basic home variables, but external images do outperform variables collected by in-person home assessors. A home’s value increases by .4 log points, when its neighbor’s visually predicted value increases by one log point, and more visible neighbors have a larger price impact than less visible neighbors. Homes that went through foreclosure during the 2008-09 financial crisis experienced a .04 log point decline in their appearance-related value, relative to comparable homes, suggesting that foreclosures reduced the incentives to maintain the housing stock. We do not find more depreciation of appearance in rental properties, or more upgrading of appearance by owners before resale.

Fine-Grained Visual Classification (FGVC) datasets contain small sample sizes, along with significant intra-class variation and inter-class similarity. While prior work has addressed intra-class variation using localization and segmentation techniques, inter-class similarity may also affect feature learning and reduce classification performance. In this work, we address this problem using a novel optimization procedure for the end-to-end neural network training on FGVC tasks. Our procedure, called Pairwise Confusion (PC) reduces overfitting by intentionally introducing confusion in the activations. With PC regularization, we obtain state-of-the-art performance on six of the most widely-used FGVC datasets and demonstrate improved localization ability. PC is easy to implement, does not need excessive hyperparameter tuning during training, and does not add significant overhead during test time.

Computer vision methods that quantify the perception of urban environment are increasingly being used to study the relationship between a city’s physical appearance and the behavior and health of its residents. Yet, the throughput of current methods is too limited to quantify the perception of cities across the world. To tackle this challenge, we introduce a new crowdsourced dataset containing 110,988 images from 56 cities, and 1,170,000 pairwise comparisons provided by 81,630 online volunteers along six perceptual attributes: safe, lively, boring, wealthy, depressing, and beautiful. Using this data, we train a Siamese-like convolutional neural architecture, which learns from a joint classification and ranking loss, to predict human judgments of pairwise image comparisons. Our results show that crowdsourcing combined with neural networks can produce urban perception data at the global scale.

Continuous-wave Time-of-flight (TOF) range imaging has become a commercially viable technology with many applications in computer vision and graphics. However, the depth images obtained from TOF cameras contain scene dependent errors due to multipath interference (MPI). Specifically, MPI occurs when multiple optical reflections return to a single spatial location on the imaging sensor. Many prior approaches to rectifying MPI rely on sparsity in optical reflections, which is an extreme simplification. In this paper, we correct MPI by combining the standard measurements from a TOF camera with information from direct and global light transport. We report results on both simulated experiments and physical experiments (using the Kinect sensor). Our results, evaluated against ground truth, demonstrate a quantitative improvement in depth accuracy.

New, “big” data sources allow measurement of city characteristics and outcome variables higher frequencies and finer geographic scales than ever before. However, big data will not solve large urban social science questions on its own. Big data has the most value for the study of cities when it allows measurement of the previously opaque, or when it can be coupled with exogenous shocks to people or place. We describe a number of new urban data sources and illustrate how they can be used to improve the study and function of cities. We first show how Google Street View images can be used to predict income in New York City, suggesting that similar image data can be used to map wealth and poverty in previously unmeasured areas of the developing world. We then discuss how survey techniques can be improved to better measure willingness to pay for urban amenities. Finally, we explain how Internet data is being used to improve the quality of city services.

Imaging through complex media is a well-known challenge, as scattering distorts a signal and invalidates imaging equations. For coherent imaging, the input field can be reconstructed using phase conjugation or knowledge of the complex transmission matrix. However, for incoherent light, wave interference methods are limited to small viewing angles. On the other hand, time-resolved methods do not rely on signal or object phase correlations, making them suitable for reconstructing wide-angle, larger-scale objects. Previously, a time-resolved technique was demonstrated for uniformly reflecting objects. Here, we generalize the technique to reconstruct the spatially varying reflectance of shapes hidden by angle-dependent diffuse layers. The technique is a noninvasive method of imaging three-dimensional objects without relying on coherence. For a given diffuser, ultrafast measurements are used in a convex optimization program to reconstruct a wide-angle, three-dimensional reflectance function. The method has potential use for biological imaging and material characterization.

Social science literature has shown a strong connection between the visual appearance of a city's neighborhoods and the behavior and health of its citizens. Yet, this research is limited by the lack of methods that can be used to quantify the appearance of streetscapes across cities or at high enough spatial resolutions. In this paper, we describe 'Streetscore', a scene understanding algorithm that predicts the perceived safety of a streetscape, using training data from an online survey with contributions from more than 7000 participants. We first study the predictive power of commonly used image features using support vector regression, finding that Geometric Texton and Color Histograms along with GIST are the best performers when it comes to predict the perceived safety of a streetscape. Using Streetscore, we create high resolution maps of perceived safety for 21 cities in the Northeast and Midwest of the United States at a resolution of 200 images/square mile, scoring ~1 million images from Google Streetview. These datasets should be useful for urban planners, economists and social scientists looking to explain the social and economic consequences of urban perception., MIT Media Lab Consortium, Google (Firm). Living Labs Tides Foundation

We use time-of-flight information in an iterative non-linear optimization algorithm to recover reflectance properties of a three-dimensional scene hidden behind a diffuser. We demonstrate reconstruction of wide-field images without relying on diffuser correlation properties.

This paper introduces the concept of time-of-flight reflectance estimation, and demonstrates a new technique that allows a camera to rapidly acquire reflectance properties of objects from a single view-point, over relatively long distances and without encircling equipment. We measure material properties by indirectly illuminating an object by a laser source, and observing its reflected light indirectly using a time-of-flight camera. The configuration collectively acquires dense angular, but low spatial sampling, within a limited solid angle range - all from a single viewpoint. Our ultra-fast imaging approach captures space-time "streak images" that can separate out different bounces of light based on path length. Entanglements arise in the streak images mixing signals from multiple paths if they have the same total path length. We show how reflectances can be recovered by solving for a linear system of equations and assuming parametric material models; fitting to lower dimensional reflectance models enables us to disentangle measurements. We demonstrate proof-of-concept results of parametric reflectance models for homogeneous and discretized heterogeneous patches, both using simulation and experimental hardware. As compared to lengthy or highly calibrated BRDF acquisition techniques, we demonstrate a device that can rapidly, on the order of seconds, capture meaningful reflectance information. We expect hardware advances to improve the portability and speed of this device., National Science Foundation (U.S.) (Award CCF-0644175), National Science Foundation (U.S.) (Award CCF-0811680), National Science Foundation (U.S.) (Award IIS-1011919), Intel Corporation (PhD Fellowship), Alfred P. Sloan Foundation (Research Fellowship)

We propose a fragment based algorithm for efficient target tracking under significant scale variation and partial occlusion. In contrast, none of the previous multiple part or fragment based algorithms are both scale adaptive and robust to partial occlusion. In our algorithm, the target is divided into a number of overlapping image fragments. Their color histograms are compared with those of candidate fragments within the neighborhood in subsequent frames. The candidate fragments with maximum histogram similarity with the template fragments contribute towards selection of location of best match within that neighborhood. Its position is further calibrated by using a localization method. We implement a systematic scale adaptive tracking scheme which is robust to significant changes in the target size. Extensive experimental results based on real life sequences and test datasets are presented which demonstrate the excellent tracking accuracy achieved by our algorithm at real time.

In this paper we propose a computationally efficient scale adaptive tracking method using a hybrid color histogram matching scheme. Firstly, we report an important property of the Chi-squared measure- It outperforms Bhattacharyya measure in the task of histogram matching from a few significantly similar multimodal histograms. Also, Bhattacharyya measure performs better while selecting matches from a varied dataset. We employ these results to develop a hybrid histogram matching procedure using the two measures. This method is used for a patch matching algorithm in real time tracking. We first calculate a color histogram of the target which is then compared with histograms of patches in the neighborhood in subsequent frames using this hybrid procedure to obtain the best match. We devise a systematic scale adaptive tracking method which is robust to rapid changes in the target size. It is also robust to partial occlusion of the target. Extensive experimental proof based on real life and test datasets is presented which demonstrates the excellent tracking accuracy achieved by our algorithm at real time.

We present a novel approach for evaluation of position and orientation of geometric shapes from scattered time-resolved data. Traditionally, imaging systems treat scattering as unwanted and are designed to mitigate the effects. Instead, we show here that scattering can be exploited by implementing a system based on a femtosecond laser and a streak camera. The result is accurate estimation of object pose, which is a fundamental tool in analysis of complex scenarios and plays an important role in our understanding of physical phenomena. Here, we experimentally show that for a given geometry, a single incident illumination point yields enough information for pose estimation and tracking after multiple scattering events. Our technique can be used for single-shot imaging behind walls or through turbid media.