Search

This paper firstly presents old photo modernization using multiple references by performing stylization and enhancement in a unified manner. In order to modernize old photos, we propose a novel multi-reference-based old photo modernization (MROPM) framework consisting of a network MROPM-Net and a novel synthetic data generation scheme. MROPM-Net stylizes old photos using multiple references via photorealistic style transfer (PST) and further enhances the results to produce modern-looking images. Meanwhile, the synthetic data generation scheme trains the network to effectively utilize multiple references to perform modernization. To evaluate the performance, we propose a new old photos benchmark dataset (CHD) consisting of diverse natural indoor and outdoor scenes. Extensive experiments show that the proposed method outperforms other baselines in performing modernization on real old photos, even though no old photos were used during training. Moreover, our method can appropriately select styles from multiple references for each semantic region in the old photo to further improve the modernization performance., Comment: Accepted to CVPR 2023. Website: https://kaist-viclab.github.io/old-photo-modernization

Facial Action Units detection (FAUs) represents a fine-grained classification problem that involves identifying different units on the human face, as defined by the Facial Action Coding System. In this paper, we present a simple yet efficient Vision Transformer-based approach for addressing the task of Action Units (AU) detection in the context of Affective Behavior Analysis in-the-wild (ABAW) competition. We employ the Video Vision Transformer(ViViT) Network to capture the temporal facial change in the video. Besides, to reduce massive size of the Vision Transformers model, we replace the ViViT feature extraction layers with the CNN backbone (Regnet). Our model outperform the baseline model of ABAW 2023 challenge, with a notable 14% difference in result. Furthermore, the achieved results are comparable to those of the top three teams in the previous ABAW 2022 challenge., Comment: Will be updated

Nuclear detection, segmentation and morphometric profiling are essential in helping us further understand the relationship between histology and patient outcome. To drive innovation in this area, we setup a community-wide challenge using the largest available dataset of its kind to assess nuclear segmentation and cellular composition. Our challenge, named CoNIC, stimulated the development of reproducible algorithms for cellular recognition with real-time result inspection on public leaderboards. We conducted an extensive post-challenge analysis based on the top-performing models using 1,658 whole-slide images of colon tissue. With around 700 million detected nuclei per model, associated features were used for dysplasia grading and survival analysis, where we demonstrated that the challenge's improvement over the previous state-of-the-art led to significant boosts in downstream performance. Our findings also suggest that eosinophils and neutrophils play an important role in the tumour microevironment. We release challenge models and WSI-level results to foster the development of further methods for biomarker discovery.

Active biological molecules present a powerful, yet largely untapped, opportunity to impart autonomous regulation to materials. Because these systems can function robustly to regulate when and where chemical reactions occur, they have the ability to bring complex, life-like behavior to synthetic materials. Here, we achieve this design feat by using functionalized circadian clock proteins, KaiB and KaiC, to engineer time-dependent crosslinking of colloids. The resulting material self-assembles with programmable kinetics, producing macroscopic changes in material properties, via molecular assembly of KaiB-KaiC complexes. We show that colloid crosslinking depends strictly on the phosphorylation state of KaiC, with kinetics that are synced with KaiB-KaiC complexing. Our microscopic image analyses and computational models indicate that the stability of colloidal super-structures depends sensitively on the number of Kai complexes per colloid connection. Consistent with our model predictions, a high concentration stabilizes the material against dissolution after a robust self-assembly phase, while a low concentration allows circadian oscillation of material structure. This work introduces the concept of harnessing biological timers to control synthetic materials; and, more generally, opens the door to using protein-based reaction networks to endow synthetic systems with life-like functional properties., Comment: 5 figures + SI

The availability of training data remains a significant obstacle for the implementation of machine learning in scientific applications. In particular, estimating how a system might respond to external forcings or perturbations requires specialized labeled data or targeted simulations, which may be computationally intensive to generate at scale. In this study, we propose a novel solution to this challenge by utilizing a principle from statistical physics known as the Fluctuation-Dissipation Theorem (FDT) to discover knowledge using an AI model that can rapidly produce scenarios for different external forcings. By leveraging FDT, we are able to extract information encoded in a large dataset produced by Earth System Models, which includes 8250 years of internal climate fluctuations, to estimate the climate system's response to forcings. Our model, AiBEDO, is capable of capturing the complex, multi-timescale effects of radiation perturbations on global and regional surface climate, allowing for a substantial acceleration of the exploration of the impacts of spatially-heterogenous climate forcers. To demonstrate the utility of AiBEDO, we use the example of a climate intervention technique called Marine Cloud Brightening, with the ultimate goal of optimizing the spatial pattern of cloud brightening to achieve regional climate targets and prevent known climate tipping points. While we showcase the effectiveness of our approach in the context of climate science, it is generally applicable to other scientific disciplines that are limited by the extensive computational demands of domain simulation models. Source code of AiBEDO framework is made available at https://github.com/kramea/kdd_aibedo. A sample dataset is made available at https://doi.org/10.5281/zenodo.7597027. Additional data available upon request.

Generic event boundary detection (GEBD) aims to split video into chunks at a broad and diverse set of actions as humans naturally perceive event boundaries. In this study, we present an approach that considers the correlation between neighbor frames with pyramid feature maps in both spatial and temporal dimensions to construct a framework for localizing generic events in video. The features at multiple spatial dimensions of a pre-trained ResNet-50 are exploited with different views in the temporal dimension to form a temporal pyramid feature map. Based on that, the similarity between neighbor frames is calculated and projected to build a temporal pyramid similarity feature vector. A decoder with 1D convolution operations is used to decode these similarities to a new representation that incorporates their temporal relationship for later boundary score estimation. Extensive experiments conducted on the GEBD benchmark dataset show the effectiveness of our system and its variations, in which we outperformed the state-of-the-art approaches. Additional experiments on TAPOS dataset, which contains long-form videos with Olympic sport actions, demonstrated the effectiveness of our study compared to others.

Object compositing based on 2D images is a challenging problem since it typically involves multiple processing stages such as color harmonization, geometry correction and shadow generation to generate realistic results. Furthermore, annotating training data pairs for compositing requires substantial manual effort from professionals, and is hardly scalable. Thus, with the recent advances in generative models, in this work, we propose a self-supervised framework for object compositing by leveraging the power of conditional diffusion models. Our framework can hollistically address the object compositing task in a unified model, transforming the viewpoint, geometry, color and shadow of the generated object while requiring no manual labeling. To preserve the input object's characteristics, we introduce a content adaptor that helps to maintain categorical semantics and object appearance. A data augmentation method is further adopted to improve the fidelity of the generator. Our method outperforms relevant baselines in both realism and faithfulness of the synthesized result images in a user study on various real-world images.

Redox monolayers are the base for a wide variety of devices including high-frequency molecular diodes or biomolecular sensors. We introduce a formalism to describe the electrochemical shot-noise of such monolayer, confirmed experimentally at room temperature in liquid. The proposed method, carried out at equilibrium, avoids parasitic capacitance, increases the sensitivity and allows to obtain quantitative information such as the electronic coupling (or standard electron transfer rates), its dispersion and the number of molecules. Unlike in solid-state physics, the homogeneity in energy levels and transfer rates in the monolayer yields a Lorentzian spectrum. This first step for shot-noise studies in molecular electrochemical systems opens perspectives for quantum transport studies in liquid environment at room temperature as well as highly sensitive measurements for bioelectrochemical sensors., Comment: 6 pages, 4 Figures (main) + 15 pages, 11 Figures (SM)

The ACII Affective Vocal Bursts (A-VB) competition introduces a new topic in affective computing, which is understanding emotional expression using the non-verbal sound of humans. We are familiar with emotion recognition via verbal vocal or facial expression. However, the vocal bursts such as laughs, cries, and signs, are not exploited even though they are very informative for behavior analysis. The A-VB competition comprises four tasks that explore non-verbal information in different spaces. This technical report describes the method and the result of SclabCNU Team for the tasks of the challenge. We achieved promising results compared to the baseline model provided by the organizers.

Numerous researchers have studied innovations in future sixth-generation (6G) wireless communications. Indeed, a critical issue that has emerged is to contend with society's insatiable demand for high data rates and massive 6G connectivity. Some scholars consider one innovation to be a breakthrough--the application of free-space optical (FSO) communication. Owing to its exceedingly high carrier frequency/bandwidth and the potential of the unlicensed spectrum domain, FSO communication provides an excellent opportunity to develop ultrafast data links that can be applied in a variety of 6G applications, including heterogeneous networks with enormous connectivity and wireless backhauls for cellular systems. In this study, we perform video signal transmissions via an FPGA-based FSO communication prototype to investigate the feasibility of an FSO link with a distance of up to 20~km. We use a channel emulator to reliably model turbulence, scintillation, and power attenuation of the long-range FSO channel. We use the FPGA-based real-time SDR prototype to process the transmitted and received video signals. Our study also presents the channel-generation process of a given long-distance FSO link. To enhance the link quality, we apply spatial selective filtering to suppress the background noise generated by sunlight. To measure the misalignment of the transceiver, we use sampling-based pointing, acquisition, and tracking to compensate for it by improving the signal-to-noise ratio. For the main video signal transmission testbed, we consider various environments by changing the amount of turbulence and wind speed. We demonstrate that the testbed even permits the successful transmission of ultra-high-definition (UHD: 3840 x 2160 resolution) 60 fps videos under severe turbulence and high wind speeds., Comment: 8 pages, 5 figures

This paper illustrates our submission method to the fourth Affective Behavior Analysis in-the-Wild (ABAW) Competition. The method is used for the Multi-Task Learning Challenge. Instead of using only face information, we employ full information from a provided dataset containing face and the context around the face. We utilized the InceptionNet V3 model to extract deep features then we applied the attention mechanism to refine the features. After that, we put those features into the transformer block and multi-layer perceptron networks to get the final multiple kinds of emotion. Our model predicts arousal and valence, classifies the emotional expression and estimates the action units simultaneously. The proposed system achieves the performance of 0.917 on the MTL Challenge validation dataset.

Facial behavior analysis is a broad topic with various categories such as facial emotion recognition, age, and gender recognition. Many studies focus on individual tasks while the multi-task learning approach is still an open research issue and requires more research. In this paper, we present our solution and experiment result for the Multi-Task Learning challenge of the Affective Behavior Analysis in-the-wild competition. The challenge is a combination of three tasks: action unit detection, facial expression recognition, and valance-arousal estimation. To address this challenge, we introduce a cross-attentive module to improve multi-task learning performance. Additionally, a facial graph is applied to capture the association among action units. As a result, we achieve the evaluation measure of 128.8 on the validation data provided by the organizers, which outperforms the baseline result of 30.

Depth maps are used in a wide range of applications from 3D rendering to 2D image effects such as Bokeh. However, those predicted by single image depth estimation (SIDE) models often fail to capture isolated holes in objects and/or have inaccurate boundary regions. Meanwhile, high-quality masks are much easier to obtain, using commercial auto-masking tools or off-the-shelf methods of segmentation and matting or even by manual editing. Hence, in this paper, we formulate a novel problem of mask-guided depth refinement that utilizes a generic mask to refine the depth prediction of SIDE models. Our framework performs layered refinement and inpainting/outpainting, decomposing the depth map into two separate layers signified by the mask and the inverse mask. As datasets with both depth and mask annotations are scarce, we propose a self-supervised learning scheme that uses arbitrary masks and RGB-D datasets. We empirically show that our method is robust to different types of masks and initial depth predictions, accurately refining depth values in inner and outer mask boundary regions. We further analyze our model with an ablation study and demonstrate results on real applications. More information can be found at https://sooyekim.github.io/MaskDepth/ ., Comment: Accepted to CVPR 2022 (camera-ready version)

Breeding for new crop characteristics and adjusting management practices are critical avenues to mitigate yield loss and maintain yield stability under a changing climate. However, identifying high-performing plant traits and management options for different growing regions through traditional breeding practices and agronomic field trials is often time and resource-intensive. Mechanistic crop simulation models can serve as powerful tools to help synthesize cropping information, set breeding targets, and develop adaptation strategies to sustain food production. In this study, we develop a modeling framework for a mechanistic crop model (MAIZSIM) to run many simulations within a trait x environment x management landscape and demonstrate how such a modeling framework could be used to identify ideal trait-management combinations that maximize yield and yield stability for different agro-climate regions in the US.

Massive multiple-input multiple-output (MIMO) and full-duplex (FD) are promising candidates for achieving the spectral efficiency to meet the needs of 5G communications. One essential key to realizing practical FD massive MIMO systems is how to effectively mitigate the self-interference (SI). Conventionally, however, the performance comparison of different SI methods by reflecting the actual channel characteristics was insufficient in the literature. Accordingly, this paper presents a performance analysis of SI cancellation (SIC) methods in FD massive MIMO systems. Analytical and numerical results confirm that, in an imperfect channel-estimation case, the ergodic rates performance of the spatial suppression in the uplink outperforms those of the SI subtraction, due to the correlation between the precoder and the estimation error of the SI channel. In addition, we discuss which method performs better under different given system constraints such as uplink and downlink sum rates, the total transmit power, and the power scaling law., Comment: 15 pages, 8 figures, (under revision) submitted to IEEE Transactions on Wireless Communications

For in-building networks, the potential of transparent antennas, which are used as windows of a building, is presented in this paper. In this scenario, a transparent window antenna communicates with outdoor devices or base stations, and the indoor repeaters act as relay stations of the transparent window antenna for indoor devices. At indoor, back lobe waves of the transparent window antenna are defined as interference to in-building networks. Hence, we analyze different SIR and SINR results according to the location of an indoor repeater through 3D ray tracing system-level simulation. Furthermore, a link-level simulation through a full-duplex software-defined radio platform with the fabricated transparent antenna is presented to examine the feasibility of the transparent antenna., Comment: 2 pages, 3 figures

For computers to recognize human emotions, expression classification is an equally important problem in the human-computer interaction area. In the 3rd Affective Behavior Analysis In-The-Wild competition, the task of expression classification includes eight classes with six basic expressions of human faces from videos. In this paper, we employ a transformer mechanism to encode the robust representation from the backbone. Fusion of the robust representations plays an important role in the expression classification task. Our approach achieves 30.35\% and 28.60\% for the $F_1$ score on the validation set and the test set, respectively. This result shows the effectiveness of the proposed architecture based on the Aff-Wild2 dataset.

Human emotions recognization contributes to the development of human-computer interaction. The machines understanding human emotions in the real world will significantly contribute to life in the future. This paper will introduce the Affective Behavior Analysis in-the-wild (ABAW3) 2022 challenge. The paper focuses on solving the problem of the valence-arousal estimation and action unit detection. For valence-arousal estimation, we conducted two stages: creating new features from multimodel and temporal learning to predict valence-arousal. First, we make new features; the Gated Recurrent Unit (GRU) and Transformer are combined using a Regular Networks (RegNet) feature, which is extracted from the image. The next step is the GRU combined with Local Attention to predict valence-arousal. The Concordance Correlation Coefficient (CCC) was used to evaluate the model.

Facial Action Coding System is an approach for modeling the complexity of human emotional expression. Automatic action unit (AU) detection is a crucial research area in human-computer interaction. This paper describes our submission to the third Affective Behavior Analysis in-the-wild (ABAW) competition 2022. We proposed a method for detecting facial action units in the video. At the first stage, a lightweight CNN-based feature extractor is employed to extract the feature map from each video frame. Then, an attention module is applied to refine the attention map. The attention encoded vector is derived using a weighted sum of the feature map and the attention scores later. Finally, the sigmoid function is used at the output layer to make the prediction suitable for multi-label AUs detection. We achieved a macro F1 score of 0.48 on the ABAW challenge validation set compared to 0.39 from the baseline model.

Nuclei Identification and Counting is the most important morphological feature of cancers, especially in the colon. Many deep learning-based methods have been proposed to deal with this problem. In this work, we construct an extension of Hovernet for nuclei identification and counting to address the problem named MF-Hovernet. Our proposed model is the combination of multiple filer block to Hovernet architecture. The current result shows the efficiency of multiple filter block to improve the performance of the original Hovernet model.