Search

This paper presents a multi-modal Gaussian process latent variable model with semi-supervised label dequantization. In real-world applications, although user ratings are often attached to the content, they are roughly provided and are limited in number due to the manual provision. The roughness and small sample problems make it difficult to estimate the ratings of the content. To solve the problem, we newly derive the semi-supervised label dequantization scheme by combining the pseudo-labeling for unlabeled data based on soft-labeling and our previously proposed label dequantization scheme. This is the main contribution of this paper. By introducing the scheme into the optimization process of a Gaussian process-based generative model, we realize the multi-modal Gaussian process latent variable model with semi-supervised label dequantization, and highly accurate rating estimation becomes feasible. The effectiveness of our method is verified through several experiments using some open datasets.

The primary objective of a recommender system (RS) is to enhance user satisfaction, which serves as the gold standard for evaluation. In order to support the advancement of RS, it is crucial to study how to accurately measure user satisfaction. This paper proposes a novel evaluation framework that leverages user-centric evaluations to achieve a more precise measurement of satisfaction. User-centric evaluations involve metrics that model the user’s behavior toward the recommended ranking list. These metrics incorporate a persistence parameter that captures the user’s personality traits. However, most conventional studies in the RS field utilize a predefined persistence parameter for all users, neglecting the individual differences among users. In contrast, our work focuses on optimizing this parameter individually for each user, which, to the best of our knowledge, is the first attempt in the RS context. To demonstrate the effectiveness of our framework in measuring satisfaction, we conducted innovative subject experiments to collect satisfaction data. Through these experiments, we obtained valuable insights into user satisfaction, which we used for quantitative correlation analysis. The results of this analysis provide empirical evidence that our framework is well-suited for RS evaluation, showing a stronger alignment with user satisfaction compared to conventional approaches.

Although text-guided image manipulation approaches have demonstrated highly accurate performance for editing the appearance of images in a virtual or simple scenario, their real-world applications face significant challenges. The primary cause of these challenges is the misalignment in the distribution of training and real-world data, which leads to unstable text-guided image manipulation. In this work, we propose a novel framework called TolerantGAN and tackle the new task of real-world text-guided image manipulation independent of the training data. To achieve this, we introduce two key concepts of a border smoothly connection module (BSCM) and a manipulation direction-based attention module (MDAM). BSCM smoothens the misalignment in the distribution of training and real-world data. MDAM extracts only regions highly relevant for image manipulation and assists in reconstructing unobserved objects in the training data. For in-the-wild input images of various classes, TolerantGAN robustly outperforms the state-of-the-art methods.

There are various sentiment theories for categorizing human sentiments into several discrete sentiment categories, which means that the theory used for training sentiment prediction methods does not always match that used in the test phase. As a solution to this problem, zero-shot visual sentiment prediction methods have been proposed to predict unseen sentiments for which no images are available in the training phase. However, the training of these previous zero-shot methods relies on a single sentiment theory, which limits their ability to handle sentiments from other theories. Thus, this article proposes a more robust zero-shot visual sentiment prediction method that can handle cross-domain sentiments defined in different sentiment theories. Specifically, by focusing on the fact that sentiments are abstract concepts common to humans regardless of whether their theories are different, we incorporate knowledge distillation into our method to construct a teacher–student model that can train the implicit relationships between sentiments defined in different sentiment theories. Furthermore, to enhance sentiment discrimination capability and strengthen the implicit relationships between sentiments, we introduce a novel sentiment loss between the teacher and student models. In this way, our model becomes robust to unseen sentiments by exploiting the implicit relationships between sentiments. The contributions of this article are the introduction of knowledge distillation and a novel sentiment loss between the teacher and student models for zero-shot visual sentiment prediction, and improved performance of zero-shot visual sentiment prediction. Experiments on several open datasets demonstrate the effectiveness of the proposed method.

This article proposes a method for transferring knowledge of semantic segmentation from a labeled source domain to an unlabeled target domain without using the source-domain data. Such a problem is called source-data-free domain adaptation, in which a pre-trained source-domain model and the unlabeled target-domain data are used to transfer the label knowledge across the domains. Like most previous methods, our method uses pseudo labels for distilling and transferring the source-domain knowledge. On the basis of the pseudo-label learning, our method improves the domain adaptation performance in two innovative ways: 1) reducing the domain differences by source-data-free style transfer and 2) exploring the style diversity within the target domain by style modification. To this end, we introduce two additional modules: 1) an inter-domain style transfer module which aligns the feature statistics of the source and target domains before producing the pseudo labels thereby improving the pseudo labels' accuracy, and 2) an intra-domain style modification module which modifies the image styles within the target domain for learning intra-domain style-invariant features. Our method with the two modules outperforms previous source-data-free domain adaptation methods in two commonly used benchmarks. Moreover, our method is well compatible with the previous methods for further improvement.

Sports data analysis has significantly advanced and become an indispensable technology for planning strategy and enhancing competitiveness. In soccer, shot prediction has been realized on the basis of historical match situations, and its results contribute to the evaluation of plays and team tactics. However, traditional event prediction methods required tracking data acquired with expensive instrumentation and event stream data annotated by experts, and the benefits were limited to only some professional athletes. To tackle this problem, we propose a novel shot prediction method using soccer videos. Our method constructs a graph considering player relationships with audio and visual features as graph nodes. Specifically, by introducing players’ importance into the graph edge based on their field positions and team information, our method enables the utilization of knowledge that reflects the detailed match situation. Next, we extract latent features considering spatial–temporal interactions from the graph and predict event occurrences with uncertainty based on the probabilistic deep learning method. In comparison with several baseline methods and ablation studies using professional soccer match data, our method was confirmed to be effective as it demonstrated the highest average precision of 0.948, surpassing other methods.

Abstract Background Collecting system entry in robot-assisted partial nephrectomy may occur even in cases showing a low N factor in the R.E.N.A.L nephrometry score. Therefore, in this study, we focused on the tumor contact surface area with the adjacent renal parenchyma and attempted to construct a novel predictive model for collecting system entry. Methods Among 190 patients who underwent robot-assisted partial nephrectomy at our institution from 2015 to 2021, 94 patients with a low N factor (1–2) were analyzed. Contact surface was measured with three-dimensional imaging software and defined as the C factor, classified as C1, < 10 cm [2]; C2, ≥ 10 and

This study presents a trial analysis that uses brain activity information obtained from mice to detect rheumatoid arthritis (RA) in its presymptomatic stages. Specifically, we confirmed that F759 mice, serving as a mouse model of RA that is dependent on the inflammatory cytokine IL-6, and healthy wild-type mice can be classified on the basis of brain activity information. We clarified which brain regions are useful for the presymptomatic detection of RA. We introduced a matrix completion-based approach to handle missing brain activity information to perform the aforementioned analysis. In addition, we implemented a canonical correlation-based method capable of analyzing the relationship between various types of brain activity information. This method allowed us to accurately classify F759 and wild-type mice, thereby identifying essential features, including crucial brain regions, for the presymptomatic detection of RA. Our experiment obtained brain activity information from 15 F759 and 10 wild-type mice and analyzed the acquired data. By employing four types of classifiers, our experimental results show that the thalamus and periaqueductal gray are effective for the classification task. Furthermore, we confirmed that classification performance was maximized when seven brain regions were used, excluding the electromyogram and nucleus accumbens.

This paper proposes a multimodal Transformer model that uses time-series data to detect and predict winter road surface conditions. For detecting or predicting road surface conditions, the previous approach focuses on the cooperative use of multiple modalities as inputs, e.g., images captured by fixed-point cameras (road surface images) and auxiliary data related to road surface conditions under simple modality integration. Although such an approach achieves performance improvement compared to the method using only images or auxiliary data, there is a demand for further consideration of the way to integrate heterogeneous modalities. The proposed method realizes a more effective modality integration using a cross-attention mechanism and time-series processing. Concretely, when integrating multiple modalities, feature compensation through mutual complementation between modalities is realized through a feature integration technique based on a cross-attention mechanism, and the representational ability of the integrated features is enhanced. In addition, by introducing time-series processing for the input data across several timesteps, it is possible to consider the temporal changes in the road surface conditions. Experiments are conducted for both detection and prediction tasks using data corresponding to the current winter condition and data corresponding to a few hours after the current winter condition, respectively. The experimental results verify the effectiveness of the proposed method for both tasks. In addition to the construction of the classification model for winter road surface conditions, we first attempt to visualize the classification results, especially the prediction results, through the image style transfer model as supplemental extended experiments on image generation at the end of the paper.

In this study, we propose a novel method for algal bed region segmentation using aerial images. Accurately determining the carbon dioxide absorption capacity of coastal algae requires measurements of algal bed regions. However, conventional manual measurement methods are resource-intensive and time-consuming, which hinders the advancement of the field. To solve these problems, we propose a novel method for automatic algal bed region segmentation using aerial images. In our method, we use an advanced semantic segmentation model, a ViT adapter, and adapt it to aerial images for algal bed region segmentation. Our method demonstrates high accuracy in identifying algal bed regions in an aerial image dataset collected from Hokkaido, Japan. The experimental results for five different ecological regions show that the mean intersection over union (mIoU) and mean F-score of our method in the validation set reach 0.787 and 0.870, the IoU and F-score for the background region are 0.957 and 0.978, and the IoU and F-score for the algal bed region are 0.616 and 0.762, respectively. In particular, the mean recognition area compared with the ground truth area annotated manually is 0.861. Our study contributes to the advancement of blue carbon assessment by introducing a novel semantic segmentation-based method for identifying algal bed regions using aerial images.

In this study, we propose a classification method of expert–novice levels using a graph convolutional network (GCN) with a confidence-aware node-level attention mechanism. In classification using an attention mechanism, highlighted features may not be significant for accurate classification, thereby degrading classification performance. To address this issue, the proposed method introduces a confidence-aware node-level attention mechanism into a spatiotemporal attention GCN (STA-GCN) for the classification of expert–novice levels. Consequently, our method can contrast the attention value of each node on the basis of the confidence measure of the classification, which solves the problem of classification approaches using attention mechanisms and realizes accurate classification. Furthermore, because the expert–novice levels have ordinalities, using a classification model that considers ordinalities improves the classification performance. The proposed method involves a model that minimizes a loss function that considers the ordinalities of classes to be classified. By implementing the above approaches, the expert–novice level classification performance is improved.

In our digitally driven society, advances in software and hardware to capture video data allow extensive gathering and analysis of large datasets. This has stimulated interest in extracting information from video data, such as buildings and urban streets, to enhance understanding of the environment. Urban buildings and streets, as essential parts of cities, carry valuable information relevant to daily life. Extracting features from these elements and integrating them with technologies such as VR and AR can contribute to more intelligent and personalized urban public services. Despite its potential benefits, collecting videos of urban environments introduces challenges because of the presence of dynamic objects. The varying shape of the target building in each frame necessitates careful selection to ensure the extraction of quality features. To address this problem, we propose a novel evaluation metric that considers the video-inpainting-restoration quality and the relevance of the target object, considering minimizing areas with cars, maximizing areas with the target building, and minimizing overlapping areas. This metric extends existing video-inpainting-evaluation metrics by considering the relevance of the target object and interconnectivity between objects. We conducted experiment to validate the proposed metrics using real-world datasets from Japanese cities Sapporo and Yokohama. The experiment results demonstrate feasibility of selecting video frames conducive to building feature extraction.

The advancements in deep learning have significantly enhanced the capability of image generation models to produce images aligned with human intentions. However, training and adapting these models to new data and tasks remain challenging because of their complexity and the risk of catastrophic forgetting. This study proposes a method for addressing these challenges involving the application of class-replacement techniques within a continual learning framework. This method utilizes selective amnesia (SA) to efficiently replace existing classes with new ones while retaining crucial information. This approach improves the model’s adaptability to evolving data environments while preventing the loss of past information. We conducted a detailed evaluation of class-replacement techniques, examining their impact on the “class incremental learning” performance of models and exploring their applicability in various scenarios. The experimental results demonstrated that our proposed method could enhance the learning efficiency and long-term performance of image generation models. This study broadens the application scope of image generation technology and supports the continual improvement and adaptability of corresponding models.

This paper tackles the electron microscope image processing for rubber material discovery. In rubber material science fields, electron microscope images are used to observe the properties of materials during their development process. Hence, by analyzing the electron microscope images with the recent image processing technology, it is expected that further effective material developments can be realized. In this paper, we propose a rubber material compound and physical property image retrieval method using the rubber material electron microscope images. The aim of our method is to support material technologists to grasp the relationships between material compounds and physical properties visually and comprehensively. Our method constructs an electron microscope image space through a conditional image generation model. The generated images are used to retrieve materials with similar compounds and physical properties. By effectively using the constructed electron microscope image space, it is expected that the technologists visually and comprehensively understand the relationships of similar materials, and the advances in material developments are accelerated.

This study proposes a novel text-guided image manipulation method that introduces referring image segmentation into a generative adversarial network. The proposed text-guided image manipulation method aims to manipulate images containing multiple objects while preserving text-unrelated regions. The proposed method assigns the task of distinguishing between text-related and unrelated regions in an image to segmentation guidance based on referring image segmentation. With this architecture, the adversarial generative network can focus on generating new attributes according to the text description and reconstructing text-unrelated regions. For the challenging input images with multiple objects, the experimental results demonstrate that the proposed method outperforms conventional methods in terms of image manipulation precision.

In this study, we present a hierarchical multi-modal multi-label attribute classification model for anime illustrations using graph convolutional networks (GCNs). The focus of this study is multi-label attribute classification, as creators of anime illustrations frequently and deliberately emphasize subtle features of characters and objects. To analyze the connections between attributes, we develop a multi-modal GCN-based model that can use semantic features of anime illustrations. To create features representing the semantic information of anime illustrations, we construct a novel captioning framework by combining real-world images with their animated style transformations. In addition, because the attributes of anime illustrations are hierarchical, we introduce a loss function that considers the hierarchy of attributes to improve classification accuracy. The proposed method has two main contributions: 1) By introducing a GCN with semantic features into the multi-label attribute classification task of anime illustrations, we capture more comprehensive relationships between attributes. 2) By following certain rules to build a hierarchical structure of attributes that appear frequently in anime illustrations, we further capture subordinate relationships between attributes. In addition, we demonstrate the effectiveness of the proposed method by experiments.

Dimensionality reduction is widely used to visualize complex high-dimensional data. This study presents a novel method for effective data visualization. Previous methods depend on local distance measurements for data manifold approximation. This leads to unreliable results when a data manifold locally oscillates because of some undesirable effects, such as noise effects. In this study, we overcome this limitation by introducing a dual approximation of a data manifold. We roughly approximate a data manifold with a neighborhood graph and prune it with a global filter. This dual scheme results in local oscillation robustness and yields effective visualization with explicit global preservation. We consider a global filter based on principal component analysis frameworks and derive it with the spectral information of the original high-dimensional data. Finally, we experiment with multiple datasets to verify our method, compare its performance to that of state-of-the-art methods, and confirm the effectiveness of our novelty and results.

Question answering (QA)-based re-ranking methods for cross-modal retrieval have been recently proposed to further narrow down similar candidate images. The conventional QA-based re-ranking methods provide questions to users by analyzing candidate images, and the initial retrieval results are re-ranked based on the user's feedback. Contrary to these developments, only focusing on performance improvement makes it difficult to efficiently elicit the user's retrieval intention. To realize more useful QA-based re-ranking, considering the user interaction for eliciting the user's retrieval intention is required. In this paper, we propose a QA-based re-ranking method with considering two important factors for eliciting the user's retrieval intention: query-image relevance and recallability. Considering the query-image relevance enables to only focus on the candidate images related to the provided query text, while, focusing on the recallability enables users to easily answer the provided question. With these procedures, our method can efficiently and effectively elicit the user's retrieval intention. Experimental results using Microsoft Common Objects in Context and computationally constructed dataset including similar candidate images show that our method can improve the performance of the cross-modal retrieval methods and the QA-based re-ranking methods.

A cross-modal image retrieval that explicitly considers semantic relationships between images and texts is proposed. Most conventional cross-modal image retrieval methods retrieve the target images by directly measuring the similarities between the candidate images and query texts in a common semantic embedding space. However, such methods tend to focus on a one-to-one correspondence between a predefined image-text pair during the training phase, and other semantically similar images and texts are ignored. By considering the many-to-many correspondences between semantically similar images and texts, a common embedding space is constructed to assure semantic relationships, which allows users to accurately find more images that are related to the input query texts. Thus, in this paper, we propose a cross-modal image retrieval method that considers semantic relationships between images and texts. The proposed method calculates the similarities between texts as semantic similarities to acquire the relationships. Then, we introduce a loss function that explicitly constructs the many-to-many correspondences between semantically similar images and texts from their semantic relationships. We also propose an evaluation metric to assess whether each method can construct an embedding space considering the semantic relationships. Experimental results demonstrate that the proposed method outperforms conventional methods in terms of this newly proposed metric.

Text-guided image editing has been highlighted in the fields of computer vision and natural language processing in recent years. The approach takes an image and text prompt as input and aims to edit the image in accordance with the text prompt while preserving text-unrelated regions. The results of text-guided image editing differ depending on the way the text prompt is represented, even if it has the same meaning. It is up to the user to decide which result best matches the intended use of the edited image. This paper assumes a situation in which edited images are posted to social media and proposes a novel text-guided image editing method to help the edited images gain attention from a greater audience. In the proposed method, we apply the pre-trained text-guided image editing method and obtain multiple edited images from the multiple text prompts generated from a large language model. The proposed method leverages the novel model that predicts post scores representing engagement rates and selects one image that will gain the most attention from the audience on social media among these edited images. Subject experiments on a dataset of real Instagram posts demonstrate that the edited images of the proposed method accurately reflect the content of the text prompts and provide a positive impression to the audience on social media compared to those of previous text-guided image editing methods.

The development of healthy peri-implant soft tissues is critical to achieving the esthetic and biological success of implant restorations throughout all stages of healing and tissue maturation, starting with provisionalization. The purpose of this study was to investigate the effects of eight different implant provisional materials on human gingival fibroblasts at various stages of cell settlement by examining initial cell attachment, growth, and function. Eight different specimens—bis-acrylic 1 and 2, flowable and bulk–fill composites, self-curing acrylic 1 and 2, milled acrylic, and titanium (Ti) alloy as a control—were fabricated in rectangular plates (n = 3). The condition of human gingival fibroblasts was divided into two groups: those in direct contact with test materials (contact experiment) and those in close proximity to test materials (proximity experiment). The proximity experiment was further divided into three phases: pre-settlement, early settlement, and late settlement. A cell culture insert containing each test plate was placed into a well where the cells were pre-cultured. The number of attached cells, cell proliferation, resistance to detachment, and collagen production were evaluated. In the contact experiment, bis-acrylics and composites showed detrimental effects on cells. The number of cells attached to milled acrylic and self-curing acrylic was relatively high, being approximately 70% and 20–30%, respectively, of that on Ti alloy. There was a significant difference between self-curing acrylic 1 and 2, even with the same curing modality. The cell retention ability also varied considerably among the materials. Although the detrimental effects were mitigated in the proximity experiment compared to the contact experiment, adverse effects on cell growth and collagen production remained significant during all phases of cell settlement for bis-acrylics and flowable composite. Specifically, the early settlement phase was not sufficient to significantly mitigate the material cytotoxicity. The flowable composite was consistently more cytotoxic than the bulk–fill composite. The harmful effects of the provisional materials on gingival fibroblasts vary considerably depending on the curing modality and compositions. Pre-settlement of cells mitigated the harmful effects, implying the susceptibility to material toxicity varies depending on the progress of wound healing and tissue condition. However, cell pre-settlement was not sufficient to fully restore the fibroblastic function to the normal level. Particularly, the adverse effects of bis-acrylics and flowable composite remained significant. Milled and self-curing acrylic exhibited excellent and acceptable biocompatibility, respectively, compared to other materials.

Traffic sign recognition is a complex and challenging yet popular problem that can assist drivers on the road and reduce traffic accidents. Most existing methods for traffic sign recognition use convolutional neural networks (CNNs) and can achieve high recognition accuracy. However, these methods first require a large number of carefully crafted traffic sign datasets for the training process. Moreover, since traffic signs differ in each country and there is a variety of traffic signs, these methods need to be fine-tuned when recognizing new traffic sign categories. To address these issues, we propose a traffic sign matching method for zero-shot recognition. Our proposed method can perform traffic sign recognition without training data by directly matching the similarity of target and template traffic sign images. Our method uses the midlevel features of CNNs to obtain robust feature representations of traffic signs without additional training or fine-tuning. We discovered that midlevel features improve the accuracy of zero-shot traffic sign recognition. The proposed method achieves promising recognition results on the German Traffic Sign Recognition Benchmark open dataset and a real-world dataset taken from Sapporo City, Japan.

At present, text-guided image manipulation is a notable subject of study in the vision and language field. Given an image and text as inputs, these methods aim to manipulate the image according to the text, while preserving text-irrelevant regions. Although there has been extensive research to improve the versatility and performance of text-guided image manipulation, research on its performance evaluation is inadequate. This study proposes Manipulation Direction (MD), a logical and robust metric, which evaluates the performance of text-guided image manipulation by focusing on changes between image and text modalities. Specifically, we define MD as the consistency of changes between images and texts occurring before and after manipulation. By using MD to evaluate the performance of text-guided image manipulation, we can comprehensively evaluate how an image has changed before and after the image manipulation and whether this change agrees with the text. Extensive experiments on Multi-Modal-CelebA-HQ and Caltech-UCSD Birds confirmed that there was an impressive correlation between our calculated MD scores and subjective scores for the manipulated images compared to the existing metrics.

Soft tissue adhesion and sealing around dental and maxillofacial implants, related prosthetic components, and crowns are a clinical imperative to prevent adverse outcomes of periodontitis and periimplantitis. Zirconia is often used to fabricate implant components and crowns. Here, we hypothesized that UV treatment of zirconia would induce unique behaviors in fibroblasts that favor the establishment of a soft tissue seal. Human oral fibroblasts were cultured on zirconia specimens to confluency before placing a second zirconia specimen (either untreated or treated with one minute of 172 nm vacuum UV (VUV) light) next to the first specimen separated by a gap of 150 µm. After seven days of culture, fibroblasts only transmigrated onto VUV-treated zirconia, forming a 2.36 mm volume zone and 5.30 mm leading edge. Cells migrating on VUV-treated zirconia were enlarged, with robust formation of multidirectional cytoplastic projections, even on day seven. Fibroblasts were also cultured on horizontally placed and 45° and 60° tilted zirconia specimens, with the latter configurations compromising initial attachment and proliferation. However, VUV treatment of zirconia mitigated the negative impact of tilting, with higher tilt angles increasing the difference in cellular behavior between control and VUV-treated specimens. Fibroblast size, perimeter, and diameter on day seven were greater than on day one exclusively on VUV-treated zirconia. VUV treatment reduced surface elemental carbon and induced superhydrophilicity, confirming the removal of the hydrocarbon pellicle. Similar effects of VUV treatment were observed on glazed zirconia specimens with silica surfaces. One-minute VUV photofunctionalization of zirconia and silica therefore promotes human oral fibroblast attachment and proliferation, especially under challenging culture conditions, and induces specimen-to-specimen transmigration and sustainable photofunctionalization for at least seven days.

The exact mechanisms by which implant surface properties govern osseointegration are incompletely understood. To gain insights into this process, we examined alterations in protein and blood recruitment around screw implants with different surface topographies and wettability using a computational fluid dynamics (CFD) model. Compared with a smooth surface, a microrough implant surface reduced protein infiltration from the outer zone to the implant thread and interface zones by over two-fold. However, the microrough implant surface slowed blood flow in the interface zone by four-fold. As a result, compared with the smooth surface, the microrough surface doubled the protein recruitment/retention index, defined as the mass of proteins present in the area per unit time. Converting implant surfaces from hydrophobic to superhydrophilic increased the mass of protein infiltration 2–3 times and slowed down blood flow by up to two-fold in the implant vicinity for both smooth and microrough surfaces. The protein recruitment/retention index was highest at the implant interface when the implant surface was superhydrophilic and microrough. Thus, this study demonstrates distinct control of the mass and speed of protein and blood flow through implant surface topography, wettability, and their combination, significantly altering the efficiency of protein recruitment. Although microrough surfaces showed both positive and negative impacts on protein recruitment over smooth surfaces, superhydrophilicity was consistently positive regardless of surface topography.

Hydrophilicity/hydrophobicity—or wettability—is a key surface characterization metric for titanium used in dental and orthopedic implants. However, the effects of hydrophilicity/hydrophobicity on biological capability remain uncertain, and the relationships between surface wettability and other surface parameters, such as topography and chemistry, are poorly understood. The objective of this study was to identify determinants of surface wettability of titanium and establish the reliability and validity of the assessment. Wettability was evaluated as the contact angle of ddH2O. The age of titanium specimens significantly affected the contact angle, with acid-etched, microrough titanium surfaces becoming superhydrophilic immediately after surface processing, hydrophobic after 7 days, and hydrorepellent after 90 days. Similar age-related loss of hydrophilicity was also confirmed on sandblasted supra-micron rough surfaces so, regardless of surface topography, titanium surfaces eventually become hydrophobic or hydrorepellent with time. On age-standardized titanium, surface roughness increased the contact angle and hydrophobicity. UV treatment of titanium regenerated the superhydrophilicity regardless of age or surface roughness, with rougher surfaces becoming more superhydrophilic than machined surfaces after UV treatment. Conditioning titanium surfaces by autoclaving increased the hydrophobicity of already-hydrophobic surfaces, whereas conditioning with 70% alcohol and hydrating with water or saline attenuated pre-existing hydrophobicity. Conversely, when titanium surfaces were superhydrophilic like UV-treated ones, autoclaving and alcohol cleaning turned the surfaces hydrorepellent and hydrophobic, respectively. UV treatment recovered hydrophilicity without exception. In conclusion, surface roughness accentuates existing wettability and can either increase or decrease the contact angle. Titanium must be age-standardized when evaluating surface wettability. Surface conditioning techniques significantly but unpredictably affect existing wettability. These implied that titanium wettability is significantly influenced by the hydrocarbon pellicle and other contaminants inevitably accumulated. UV treatment may be an effective strategy to standardize wettability by making all titanium surfaces superhydrophilic, thereby allowing the characterization of individual surface topography and chemistry parameters in future studies.

In this paper, we present refining graph representation for cross-domain recommendation (CDR) based on edge pruning considering feature distribution in a latent space. Conventional graph-based CDR methods have utilized all ratings and purchase histories of user’s products. However, some items purchased by users are not related to the domain for recommendation, and this information becomes noise when making CDR. So, the proposed method introduces edge pruning into the latest graph-based CDR method to refine graph representation. To compare the item embedding features calculated in different domains, we construct a latent space and perform edge pruning through their correlations. Additionally, we introduce a state-of-the-art graph neural network into the graph construction of the proposed method that considers the interactions between users and items thereby obtaining effective embedding features in a domain. This makes it possible to consider domain-specific user preferences and estimate embedding features with high-expressive power. Furthermore, to compare the embedding features of items in the two domains, we construct their latent spaces and project them. Edge pruning is performed using the correlation of items between the two domains on the latent space. We obtain cross domain specific graph representation through edge pruning, which improves the performance by considering the relationship between both items across domains. To the best of our knowledge, no study in the CDR field focuses on eliminating unnecessary node information. We have demonstrated the effectiveness of the proposed method by comparing several graph-based state-of-the-art methods.

The mechanisms underlying bone-implant integration, or osseointegration, are still incompletely understood, in particular how blood and proteins are recruited to implant surfaces. The objective of this study was to visualize and quantify the flow of blood and the model protein fibrinogen using a computational fluid dynamics (CFD) implant model. Implants with screws were designed with three different surface topographies: (1) amorphous, (2) nano-trabecular, and (3) hybrid meso-spikes and nano-trabeculae. The implant with nano-topography recruited more blood and fibrinogen to the implant interface than the amorphous implant. Implants with hybrid topography further increased recruitment, with particularly efficient recruitment from the thread area to the interface. Blood movement significantly slowed at the implant interface compared with the thread area for all implants. The blood velocity at the interface was 3- and 4-fold lower for the hybrid topography compared with the nano-topography and amorphous surfaces, respectively. Thus, this study for the first time provides insights into how different implant surfaces regulate blood dynamics and the potential advantages of surface texturization in blood and protein recruitment and retention. In particular, co-texturization with a hybrid meso- and nano-topography created the most favorable microenvironment. The established CFD model is simple, low-cost, and expected to be useful for a wide range of studies designing and optimizing implants at the macro and micro levels.

Zero-shot neural decoding aims to decode image categories, which were not previously trained, from functional magnetic resonance imaging (fMRI) activity evoked when a person views images. However, having insufficient training data due to the difficulty in collecting fMRI data causes poor generalization capability. Thus, models suffer from the projection domain shift problem when novel target categories are decoded. In this paper, we propose a zero-shot neural decoding approach with semi-supervised multi-view embedding. We introduce the semi-supervised approach that utilizes additional images related to the target categories without fMRI activity patterns. Furthermore, we project fMRI activity patterns into a multi-view embedding space, i.e., visual and semantic feature spaces of viewed images to effectively exploit the complementary information. We define several source and target groups whose image categories are very different and verify the zero-shot neural decoding performance. The experimental results demonstrate that the proposed approach rectifies the projection domain shift problem and outperforms existing methods.

Dimensionality reduction plays an important role in interpreting and visualizing high-dimensional data. Previous methods for data visualization overestimate the local structure and lack the consideration of global preservation. In this study, we develop a Gaussian process latent variable model (GP-LVM) for data visualization. GP-LVMs are one of the frameworks of principal component analysis and preserve the global structure effectively. The drawbacks of GP-LVMs are the absence of local structure preservation and the use of low-expressive kernel functions. Therefore, we introduce regularization for local preservation and an expressive kernel function into GP-LVMs to overcome these limitations. As a result, we reflect the global and local structures in low-dimensional representations, improving the reliability and visibility of embeddings. We conduct qualitative and quantitative experiments comparing baselines and state-of-the-art methods on image and text datasets.

Abstract. Early mobilization decreases the likelihood of negative outcomes for acute-phase inpatients. Adverse events occurring during intensive care unit rehabilitation have previously been reported; however, no study has reported the incidence rates for adverse events during the acute rehabilitation phase. This study aimed to investigate the incidence of severe adverse events during acute-phase rehabilitation and evaluate them in detail. Reports of adverse events occurring during acute-phase rehabilitation in a university hospital from April 1, 2011 to March 31, 2018 were retrospectively assessed. Nine severe adverse events occurred during this period (incidence rate, 0.032%), comprising 2 cardiopulmonary arrests, 2 pulseless electrical activity events, 2 deterioration in consciousness events, 1 deterioration in consciousness event due to cerebral infarction, 1 fracture due to a fall, and 1 event involving removal of a ventricular drain. Pulmonary thromboembolism was implicated in 1 adverse event involving pulseless electrical activity and 1 deterioration in consciousness event. The causes for the 6 other adverse events could not be identified. The mean days from admission and the onset of rehabilitation to adverse event occurrence were 22.0 ± 18.2 and 17.9 ± 13.5 days (mean ± standard deviation), respectively. Four of 9 patients died, and 5 patients were discharged home or transferred to other stepdown facilities. When assessed retrospectively, there were no conflicts between patient conditions and the cancellation criteria of rehabilitation by the Japanese Association of Rehabilitation Medicine. The occurrences of severe adverse event may not be related to early mobilization (or onset time of rehabilitation) and compliance status of cancellation criteria.

In this paper, we propose a hierarchical multi-modal multi-label attribute classification model for anime illustrations using a graph convolutional network (GCN). Our focus is on the challenging task of multi-label attribute classification, which requires capturing subtle features intentionally highlighted by creators of anime illustrations. To address the hierarchical nature of these attributes, we leverage hierarchical clustering and hierarchical label assignments to organize the attribute information into a hierarchical feature. The proposed GCN-based model effectively utilizes this hierarchical feature to achieve high accuracy in multi-label attribute classification. The contributions of the proposed method are as follows. Firstly, we introduce GCN to the multi-label attribute classification task of anime illustrations, enabling the capturing of more comprehensive relationships between attributes from their co-occurrence. Secondly, we capture subordinate relationships among the attributes by adopting hierarchical clustering and hierarchical label assignment. Lastly, we construct a hierarchical structure of attributes that appear more frequently in anime illustrations based on certain rules derived from previous studies, which helps to reflect the relationships between different attributes. The experimental results on multiple datasets show that the proposed method is effective and extensible by comparing it with some existing methods, including the state-of-the-art method.

Content-based image retrieval (CBIR) aims to find desired images similar to the image input by the user, and it is extensively used in the real world. Conventional CBIR methods do not consider user preferences since they only determine retrieval results by referring to the degree of resemblance or likeness between the query and potential candidate images. Because of the above reason, a “semantic gap” appears, as the model may not accurately understand the potential intention that a user has included in the query image. In this article, we propose a re-ranking method for CBIR that considers a user’s gaze trace as interactive information to help the model predict the user’s inherent attention. The proposed method uses the user’s gaze trace corresponding to the image obtained from the initial retrieval as the user’s preference information. We introduce image captioning to effectively express the relationship between images and gaze information by generating image captions based on the gaze trace. As a result, we can transform the coordinate data into a text format and explicitly express the semantic information of the images. Finally, image retrieval is performed again using the generated gaze-dependent image captions to obtain images that align more accurately with the user’s preferences or interests. The experimental results on an open image dataset with corresponding gaze traces and human-generated descriptions demonstrate the efficacy or efficiency of the proposed method. Our method considers visual information as the user’s feedback to achieve user-oriented image retrieval.

In soccer, quantitatively evaluating the performance of players and teams is essential to improve tactical coaching and players’ decision-making abilities. To achieve this, some methods use predicted probabilities of shoot event occurrences to quantify player performances, but conventional shoot prediction models have not performed well and have failed to consider the reliability of the event probability. This paper proposes a novel method that effectively utilizes players’ spatio-temporal relations and prediction uncertainty to predict shoot event occurrences with greater accuracy and robustness. Specifically, we represent players’ relations as a complete bipartite graph, which effectively incorporates soccer domain knowledge, and capture latent features by applying a graph convolutional recurrent neural network (GCRNN) to the constructed graph. Our model utilizes a Bayesian neural network to predict the probability of shoot event occurrence, considering spatio-temporal relations between players and prediction uncertainty. In our experiments, we confirmed that the proposed method outperformed several other methods in terms of prediction performance, and we found that considering players’ distances significantly affects the prediction accuracy.

This study proposes a novel off-screen sound separation method based on audio-visual pre-training. In the field of audio-visual analysis, researchers have leveraged visual information for audio manipulation tasks, such as sound source separation. Although such audio manipulation tasks are based on correspondences between audio and video, these correspondences are not always established. Specifically, sounds coming from outside a screen have no audio-visual correspondences and thus interfere with conventional audio-visual learning. The proposed method separates such off-screen sounds based on their arrival directions using binaural audio, which provides us with three-dimensional sensation. Furthermore, we propose a new pre-training method that can consider the off-screen space and use the obtained representation to improve off-screen sound separation. Consequently, the proposed method can separate off-screen sounds irrespective of the direction from which they arrive. We conducted our evaluation using generated video data to circumvent the problem of difficulty in collecting ground truth for off-screen sounds. We confirmed the effectiveness of our methods through off-screen sound detection and separation tasks.

Abstract Background To improve the diagnostic accuracy of axillary lymph node (LN) metastasis in breast cancer patients using 2-[18F]FDG-PET/CT, we constructed an artificial intelligence (AI)-assisted diagnosis system that uses deep-learning technologies. Materials and methods Two clinicians and the new AI system retrospectively analyzed and diagnosed 414 axillae of 407 patients with biopsy-proven breast cancer who had undergone 2-[18F]FDG-PET/CT before a mastectomy or breast-conserving surgery with a sentinel lymph node (LN) biopsy and/or axillary LN dissection. We designed and trained a deep 3D convolutional neural network (CNN) as the AI model. The diagnoses from the clinicians were blended with the diagnoses from the AI model to improve the diagnostic accuracy. Results Although the AI model did not outperform the clinicians, the diagnostic accuracies of the clinicians were considerably improved by collaborating with the AI model: the two clinicians' sensitivities of 59.8% and 57.4% increased to 68.6% and 64.2%, respectively, whereas the clinicians' specificities of 99.0% and 99.5% remained unchanged. Conclusions It is expected that AI using deep-learning technologies will be useful in diagnosing axillary LN metastasis using 2-[18F]FDG-PET/CT. Even if the diagnostic performance of AI is not better than that of clinicians, taking AI diagnoses into consideration may positively impact the overall diagnostic accuracy.

Decoding a person's cognitive contents from evoked brain activity is becoming important in the field of brain-computer interaction. Previous studies have decoded a perceived image from functional magnetic resonance imaging (fMRI) activity by constructing brain decoding models that were trained with a single subject's fMRI data. However, accurate decoding is still challenging since fMRI data acquired from only a single subject have several disadvantageous characteristics such as small sample size, noisy nature, and high dimensionality. In this article, we propose a method to decode categories of perceived images from fMRI activity using shared information of multi-subject fMRI data. Specifically, by aggregating fMRI data of multiple subjects that contain a large number of samples, we extract a low-dimensional latent representation shared by multi-subject fMRI data. Then the latent representation is nonlinearly transformed into visual features and semantic features of the perceived images to identify categories from various candidate categories. Our approach leverages rich information obtained from multi-subject fMRI data and improves the decoding performance. Experimental results obtained by using two public fMRI datasets showed that the proposed method can more accurately decode categories of perceived images from fMRI activity than previous approaches using a single subject's fMRI data.

Distress image retrieval for infrastructure maintenance via self-trained deep metric learning using experts’ knowledge is proposed in this paper. Since engineers take multiple images of a single distress part for inspection of road structures, it is necessary to construct a similar distress image retrieval method considering the input of multiple images to support determination of the level of deterioration. Thus, the construction of an image retrieval method while selecting an effective input from multiple images is described in this paper. The proposed method performs deep metric learning by using a small number of effective images labeled by experts’ knowledge with information about their effectiveness and a large number of unlabeled images via a self-training approach. Specifically, an end-to-end learning approach that performs retraining of the model by assigning pseudo-labels to these unlabeled images according to the output confidence of the model is achieved. Thus, the proposed method can select an effective image from multiple images that are input at the retrieval as a query image. This is the main contribution of this paper. As a result, the proposed method realizes highly accurate retrieval of similar distress images considering the actual situation of inspection in which multiple images of a distress part are input.

A novel method for detection of important scenes in baseball videos based on correlation maximization between heterogeneous modalities via bidirectional time lag aware deep multiset canonical correlation analysis (BiTl-dMCCA) is presented in this paper. The proposed method enables detection of important scenes by collaboratively using baseball videos and their corresponding tweets. The technical contributions of this paper are twofold. First, since there are time lags between not only “tweets and corresponding multiple previous events” but also “events and corresponding multiple following posted tweets”, the proposed method considers these bidirectional time lags. Specifically, the representation of such bidirectional time lags into the derivation of their covariance matrices is newly introduced. Second, the proposed method adopts textual, visual and audio features calculated from tweets and videos as multi-modal time series features. Important scenes are detected as abnormal scenes via anomaly detection based on a generative adversarial network using multi-modal features projected by BiTl-dMCCA. The proposed method does not need any training data with annotation. Experimental results obtained by applying the proposed method to actual baseball matches show the effectiveness of the proposed method.

We propose a novel method that can learn easy-to-interpret latent representations in real-world image datasets using a VAE-based model by splitting an image into several disjoint regions. Our method performs object-wise disentanglement by exploiting image segmentation and alpha compositing. With remarkable results obtained by unsupervised disentanglement methods for toy datasets, recent studies have tackled challenging disentanglement for real-world image datasets. However, these methods involve deviations from the standard VAE architecture, which has favorable disentanglement properties. Thus, for disentanglement in images of real-world image datasets with preservation of the VAE backbone, we designed an encoder and a decoder that embed an image into disjoint sets of latent variables corresponding to objects. The encoder includes a pre-trained image segmentation network, which allows our model to focus only on representation learning while adopting image segmentation as an inductive bias. Evaluations using real-world image datasets, CelebA and Stanford Cars, showed that our method achieves improved disentanglement and transferability.

This study presents a novel feature integration method for interest level estimation using a semi-supervised multimodal Gaussian process latent variable model with pseudo-labels (semi-MGPPL). Semi-MGPPL is an extended version of the multimodal Gaussian process latent variable model (mGPLVM). It integrates features calculated from multiple modalities to predict the users’ interest levels in content. It is known that reflecting known interest levels of known users in the latent space effectively improves the accuracy of interest level estimation. However, previous methods have difficulty reflecting the interest levels when the number of samples is insufficient. Semi-MGPPL efficiently reflects interest levels in the latent space by pseudo-labeling of unlabeled samples and increasing the number of available pairs among labeled samples. In addition, obtaining behavior features is difficult for a new test sample. However, requirement of features of all modalities by previous mGPLVM-based methods makes the calculation of latent variables of a test sample challenging. Semi-MGPPL solves this problem by training a projection function from the original feature to the latent space. The experimental results on real data demonstrate the effectiveness and robustness of semi-MGPPL.

Purpose This study aimed to analyze technical and clinical factors related to oncological outcomes in patients with localized prostate cancer (PC) who were treated with whole-gland high-intensity focused ultrasound (HIFU). Materials and Methods From 2007–2014, patients diagnosed with localized PC who underwent whole-gland HIFU were consecutively included retrospectively. Biochemical failure was defined according to the Phoenix ASTRO guidelines. The relationship between oncological outcomes and technical and clinical factors was evaluated. Results The study cohort included 428 patients. The median age was 67 years, and the median prostate-specific antigen level was 7.61 ng/mL. Patient risk classifications were low (n = 102), intermediate (n = 240), and high (n = 86). Biochemical disease-free survival rates of patients with HIFU for localized PC in the total, low-, intermediate-, and high-risk groups according to D’Amico risk groups over a median follow-up period of 5 years (range 9-144) were 68.4%, 80.4%, 65.6%, and 61.6%, respectively. In multivariate logistic regression analyses to predict biochemical failure of the treatment, neoadjuvant hormonal therapy (NHT) in the high-risk group (OR 0.225, p = 0.015), and compression method in the low- (OR 0.178, p = 0.030), intermediate- (OR0.291, p

A new style transfer-based image manipulation framework combining generative networks and style transfer networks is presented in this paper. Unlike conventional style transfer tasks, we tackle a new task, text-guided image manipulation. We realize style transfer-based image manipulation that does not require any reference style images and generate a style image from the user’s input sentence. In our method, since an initial reference input sentence for a content image can automatically be given by an image-to-text model, the user only needs to update the reference sentence. This scheme can help users when they do not have any images representing the desired style. Although this text-guided image manipulation is a new challenging task, quantitative and qualitative comparisons showed the superiority of our method.

Supervised fractional-order embedding multiview canonical correlation analysis via ordinal label dequantization (SFEMCCA-OLD) for image interest estimation is presented in this paper. SFEMCCA-OLD is a CCA method that realizes accurate integration of features including low-dimensional ordinal label features. In general, since information is lost due to a limitation of the number of classes, i.e., the dimension of ordinal label information is smaller than those of other features, derivation of highly accurate integration of features is difficult. In SFEMCCA-OLD, the dimension of the ordinal label information can be increased by estimation of the canonical correlation between multiview features. We call this approach ordinal label dequantization. In addition, by introducing a fractional-order technique, our method can calculate optimal projections for noisy data such as real data. Experimental results show that the accuracy of SFEMCCA-OLD for image interest estimation is better than that of recent CCA-based methods.

A novel trial for estimating popularity of artists in music streaming services (MSS) is presented in this paper. The main contribution of this paper is to improve extensibility for using multi-modal features to accurately analyze latent relationships between artists. In the proposed method, a novel framework to construct a network is derived by collaboratively using social metadata and multi-modal features via canonical correlation analysis. Different from conventional methods that do not use multi-modal features, the proposed method can construct a network that can capture social metadata and multi-modal features, i.e., a context-aware network. For effectively analyzing the context-aware network, a novel framework to realize popularity estimation of artists is developed based on network analysis. The proposed method enables effective utilization of the network structure by extracting node features via a node embedding algorithm. By constructing an estimator that can distinguish differences between the node features, the proposed method can archive accurate popularity estimation of artists. Experimental results using multiple real-world datasets that contain artists in various genres in Spotify, one of the largest MSS, are presented. Quantitative and qualitative evaluations show that our method is effective for both classifying and regressing the popularity.

A new approach that drastically improves cross-modal retrieval performance in vision and language (hereinafter referred to as “vision and language retrieval”) is proposed in this paper. Vision and language retrieval takes data of one modality as a query to retrieve relevant data of another modality, and it enables flexible retrieval across different modalities. Most of the existing methods learn optimal embeddings of visual and lingual information to a single common representation space. However, we argue that the forced embedding optimization results in loss of key information for sentences and images. In this paper, we propose an effective utilization of representation spaces in a simple but robust vision and language retrieval method. The proposed method makes use of multiple individual representation spaces through text-to-image and image-to-text models. Experimental results showed that the proposed approach enhances the performance of existing methods that embed visual and lingual information to a single common representation space.

Techniques for integrating different types of multiple features effectively have been actively studied in recent years. Multiset canonical correlation analysis (MCCA), which maximizes the sum of pairwise correlations of inter-view (i.e., between different features), is one of the powerful methods for integrating different types of multiple features, and various MCCA-based methods have been proposed. This work focuses on a supervised MCCA variant in order to construct a novel effective feature integration framework. In this paper, we newly propose supervised fractional-order embedding geometrical multi-view CCA (SFGMCCA). This method constructs not only the correlation structure but also two types of geometrical structures of intra-view (i.e., within each feature) and inter-view simultaneously, thereby realizing more precise feature integration. This method also supports the integration of small sample and high-dimensional data by using the fractional-order technique. We conducted experiments using four types of image datasets, i.e., MNIST, COIL-20, ETH-80 and CIFAR-10. Furthermore, we also performed an fMRI dataset containing brain signals to verify the robustness. As a result, it was confirmed that accuracy improvements using SFGMCCA were statistically significant at the significance level of 0.05 compared to those using conventional representative MCCA-based methods.

A method for estimating interest levels from behavior features via tensor completion including adaptive similar user selection is presented in this paper. The proposed method focuses on a tensor that is suitable for data containing multiple contexts and constructs a third-order tensor in which three modes are “products”, “users” and “user behaviors and interest levels” for these products. By complementing this tensor, unknown interest level estimation of a product for a target user becomes feasible. For further improving the estimation performance, the proposed method adaptively selects similar users for the target user by focusing on converged estimation errors between estimated interest levels and known interest levels in the tensor completion. Furthermore, the proposed method can adaptively estimate the unknown interest from the similar users. This is the main contribution of this paper. Therefore, the influence of users having different interests is reduced, and accurate interest level estimation can be realized. In order to verify the effectiveness of the proposed method, we show experimental results obtained by estimating interest levels of users holding books.

A human-centric emotion estimation method based on correlation maximization with consideration of changes with time in visual attention and brain activity when viewing images is proposed in this paper. Owing to the recent developments of many kinds of biological sensors, many researchers have focused on multimodal emotion estimation using both eye gaze data and brain activity data for improving the quality of emotion estimation. In this paper, a novel method that focuses on the following two points is introduced. First, in order to reduce the burden on users, we obtain brain activity data from users only in the training phase by using a projection matrix calculated by canonical correlation analysis (CCA) between gaze-based visual features and brain activity-based features. Next, for considering the changes with time in both visual attention and brain activity, we obtain novel features based on CCA-based projection in each time unit. In order to include these two points, the proposed method analyzes a fourth-order gaze and image tensor for which modes are pixel location, color channel and the changes with time in visual attention. Moreover, in each time unit, the proposed method performs CCA between gaze-based visual features and brain activity-based features to realize human-centric emotion estimation with a high level of accuracy. Experimental results show that accurate human emotion estimation is achieved by using our new human-centric image representation.

Recently, it was reported that children recovering from coronavirus disease (COVID-19) developed multisystem inflammatory syndrome in children (MIS-C), which causes severe inflammation in multiple organs of the body. Because MIS-C is a new disease, the pathophysiology and prognosis are unknown. Owing to a lack of studies on this subject, we herein provide information on rehabilitation for children with MIS-C. A 12-year-old male patient presented with systemic inflammatory symptoms after approximately 2 months since recovery from COVID-19. He was treated with cyclosporine and steroid pulse therapy after admission to our hospital. His general condition improved significantly within approximately 1 week. Thereafter, his lower legs turned dark purple and he experienced intense pain whenever the lower limbs hung below the heart, such as in the sitting position. The patient was referred to the rehabilitation department, as he had difficulties during standing and walking. Because the symptoms improved with elevation of the lower extremities, we considered that the pain was related to venous stasis. The pain reduced when an elastic bandage was applied for the prevention of venous stasis; therefore, exercise therapy was implemented while the patient wore the elastic bandage. The patient's lower extremity symptoms improved in 10 days. He was discharged after 16 days and could independently perform activities of daily living (ADL). The mechanism underlying the patient's pain could not be determined; however, rehabilitation was effective when combined with compression therapy using an elastic bandage.