Search

Most current multi-modal summarization methods follow a cascaded manner, where an off-the-shelf object detector is first used to extract visual features. After that, these visual features are fused with language representations for the decoder to generate the text summary. However, the cascaded way employs separate encoders for different modalities, which makes it hard to learn the joint vision and language representation. In addition, they also ignore the semantics alignment between paragraphs and images for multi-modal summarization tasks, which are crucial to a precise summary. To tackle these issues, in this paper, we propose ViL-Sum to jointly model paragraph-level Vision-Language Semantic Alignment and Multi-Modal Summarization. Our ViL-Sum contains two components for better learning multi-modal semantics and aims to align them. The first one is a joint multi-modal encoder. The other one is two well-designed tasks for multi-task learning, including image reordering and image selection. Specifically, the joint multi-modal encoder converts images into visual embeddings and attaches them with text embedding as the input of the encoder. The reordering task guides the model to learn paragraph-level semantic alignment, and the selection task guides the model to select summary-related images in the final summary. Experimental results show that our proposed ViL-Sum outperforms current state-of-the-art methods on most automatic and manual evaluation metrics. In further analysis, we find that two well-designed tasks and a joint multi-modal encoder can effectively guide the model to learn reasonable paragraph-image and summary-image relations. [ABSTRACT FROM AUTHOR]

Background: Despite improvement in treatment strategies for atrial fibrillation (AF), a significant proportion of patients still experience recurrence after ablation. This study aims to propose a novel algorithm based on Transformer using surface electrocardiogram (ECG) signals and clinical features can predict AF recurrence. Methods: Between October 2018 to December 2021, patients who underwent index radiofrequency ablation for AF with at least one standard 10-second surface ECG during sinus rhythm were enrolled. An end-to-end deep learning framework based on Transformer and a fusion module was used to predict AF recurrence using ECG and clinical features. Model performance was evaluated using areas under the receiver operating characteristic curve (AUROC), sensitivity, specificity, accuracy and F1-score. Results: A total of 920 patients (median age 61 [IQR 14] years, 66.3% male) were included. After a median follow-up of 24 months, 253 patients (27.5%) experienced AF recurrence. A single deep learning enabled ECG signals identified AF recurrence with an AUROC of 0.769, sensitivity of 75.5%, specificity of 61.1%, F1 score of 55.6% and overall accuracy of 65.2%. Combining ECG signals and clinical features increased the AUROC to 0.899, sensitivity to 81.1%, specificity to 81.7%, F1 score to 71.7%, and overall accuracy to 81.5%. Conclusions: The Transformer algorithm demonstrated excellent performance in predicting AF recurrence. Integrating ECG and clinical features enhanced the models' performance and may help identify patients at low risk for AF recurrence after index ablation. [ABSTRACT FROM AUTHOR]

Multimodal open question answering involves retrieving relevant information from both images and their corresponding texts given a question and then generating the answer. The quality of the generated answer heavily depends on the quality of the retrieved image–text pairs. Existing methods encode and retrieve images and texts, inputting the retrieved results into a language model to generate answers. These methods overlook the semantic alignment of image–text pairs within the information source, which affects the encoding and retrieval performance. Furthermore, these methods are highly dependent on retrieval performance, and poor retrieval quality can lead to poor generation performance. To address these issues, we propose a prompt-enhanced generation model, PEG, which includes generating supplementary descriptions for images to provide ample material for image–text alignment while also utilizing vision–language joint encoding to improve encoding effects and thereby enhance retrieval performance. Contrastive learning is used to enhance the model's ability to discriminate between relevant and irrelevant information sources. Moreover, we further explore the knowledge within pre-trained model parameters through prefix-tuning to generate background knowledge relevant to the questions, offering additional input for answer generation and reducing the model's dependency on retrieval performance. Experiments conducted on the WebQA and MultimodalQA datasets demonstrate that our model outperforms other baseline models in retrieval and generation performance. [ABSTRACT FROM AUTHOR]

Named entity recognition (NER) is a critical subtask in natural language processing. It is particularly valuable to gain a deeper understanding of entity boundaries and entity types when addressing the NER problem. Most previous sequential labeling models are task-specific, while recent years have witnessed the rise of generative models due to the advantage of tackling NER tasks in the encoder–decoder framework. Despite achieving promising performance, our pilot studies demonstrate that existing generative models are ineffective at detecting entity boundaries and estimating entity types. In this paper, a multiple attention framework is proposed which introduces the attention of entity-type embedding and word–word relation into the named entity recognition task. To improve the accuracy of entity-type mapping, we adopt an external knowledge base to calculate the prior entity-type distributions and then incorporate the information input to the model via the encoder's self-attention. To enhance the contextual information, we take the entity types as part of the input. Our method obtains the other attention from the hidden states of entity types and utilizes it in self- and cross-attention mechanisms in the decoder. We transform the entity boundary information in the sequence into word–word relations and extract the corresponding embedding into the cross-attention mechanism. Through word–word relation information, the method can learn and understand more entity boundary information, thereby improving its entity recognition accuracy. We performed experiments on extensive NER benchmarks, including four flat and two long entity benchmarks. Our approach significantly improves or performs similarly to the best generative NER models. The experimental results demonstrate that our method can substantially enhance the capabilities of generative NER models. [ABSTRACT FROM AUTHOR]

Automatic Exploit Generation (AEG) involves automatically discovering paths in a program that trigger vulnerabilities, thereby generating exploits. While there is considerable research on heap-related vulnerability detection, such as detecting Heap Overflow and Use After Free (UAF) vulnerabilities, among contemporary heap-automated exploit techniques, only certain automated exploit techniques can hijack program control flow to the shellcode. An important limitation of this approach is that it cannot effectively bypass Linux's protection mechanisms. To solve this problem, we introduced Automatic Advanced Heap Exploit Generation (AAHEG). It first applies symbolic execution to analyze heap-related primitives in files and then detects potential heap-related vulnerabilities without a source code. After identifying these vulnerabilities, AAHEG builds an exploit abstract syntax tree (AST) to identify one or more successful exploit strategies, such as fast bin attack and Safe-unlink. AAHEG then selects exploitable methods via an abstract syntax tree (AST) and performs final testing to produce the final exploit. AAHEG chose to generate advanced heap-related exploits because the exploits can bypass Linux protections. Basically, AAHEG can automatically detect heap-related vulnerabilities in binaries without source code, build an exploit AST, choose from a variety of advanced heap exploit methods, bypass all Linux protection mechanisms, and generate final file-form exploit based on pwntools which can pass local and remote testing. Experimental results show that AAHEG successfully completed vulnerability detection and exploit generation for 20 Capture The Flag (CTF) binary files, 11 of which have all protection mechanisms enabled. [ABSTRACT FROM AUTHOR]

Numerical information plays an important role in various fields such as scientific, financial, social, statistics, and news. Most prior studies adopt unsupervised methods by designing complex handcrafted pattern-matching rules to extract numerical information, which can be difficult to scale to the open domain. Other supervised methods require extra time, cost, and knowledge to design, understand, and annotate the training data. To address these limitations, we propose QuantityIE, a novel approach to extracting numerical information as structured representations by exploiting syntactic features of both constituency parsing (CP) and dependency parsing (DP). The extraction results may also serve as distant supervision for zero-shot model training. Our approach outperforms existing methods from two perspectives: (1) the rules are simple yet effective, and (2) the results are more self-contained. We further propose a numerical information retrieval approach based on QuantityIE to answer analytical queries. Experimental results on information extraction and retrieval demonstrate the effectiveness of QuantityIE in extracting numerical information with high fidelity. [ABSTRACT FROM AUTHOR]

We propose SP-NLG: A semantic-parsing-guided natural language generation framework for logical content generation with high fidelity. Prior studies adopt large pretrained language models and coarse-to-fine decoding techniques to generate text with logic; while achieving considerable results on automatic evaluation metrics, they still face challenges in terms of logical fidelity based on human evaluation. Inspired by semantic parsing, which translates natural language utterances into executable logical forms, we propose to guide the generation process with a semantic parser. Different from semantic parsing and QA tasks, of which the logical correctness can be evaluated based on the execution result, the logic information of the generated content is implicit. To leverage a semantic parser for generation, we propose a slot-tied back-search algorithm. We follow the coarse-to-fine generation scheme, but instead of filling the slots with model predictions, which is less uncontrolled, the slot values are offline searched by the algorithm. The slot-tied back-search algorithm ties the parameters of a logic form with the slots of a template in one-to-one correspondence. We back-search the arguments that correctly execute the logic form and fill the arguments (as slot values) into the textual template to generate the final target, which ensures the logical correctness. Experiment results of a model built on SP-NLG demonstrates that our framework achieves high fidelity on logical text generation. [ABSTRACT FROM AUTHOR]

Fake news is a longstanding issue that has existed on the social network, whose negative impact has been increasingly recognized since the US presidential election. During the election, numerous fake news about the candidates distributes vastly in the online social networks. Identifying inauthentic news quickly is an essential purpose for this research to enhance the trustworthiness of news in online social networks, which will be the task studied in this paper. The fake news stance detection can contribute to detect a startling amount of fake news, which aims at evaluating the relevance between the headline and text bodies. There exists a significant difference between news article headline and text body, since headlines with several key phrases are usually much shorter than the text bodies. Such an information imbalance challenge may cause serious problems for the stance detection task. Furthermore, news article data in online social networks is usually exposed to various types of noise and can be contaminated, which poses more challenges for the stance detection task. In this paper, we propose a novel fake news stance detection model, namely Adversarial Pseudo-Siamese Network model (APSN), to solve these challenges. With coupled input components with imbalanced parameters, APSN can learn and compute feature vectors and similarity score of news article headlines and text bodies simultaneously. In addition, by adopting adversarial setting, besides the regular training set, a set of noisy training instances will be generated and fed to APSN in the learning process, which can significantly enhance the robustness of the model. Extensive experiments have been conducted on a real-world fake news dataset, and the experimental results reveal that the presented model exceeds compared suspicious information detection models with significant advantages. [ABSTRACT FROM AUTHOR]

Due to the outstanding performance of deep neural networks (DNNs), many researchers have begun to transfer deep learning techniques to their fields. To detect algorithmically generated domains (AGDs) generated by domain generation algorithm (DGA) in botnets, a long short-term memory (LSTM)-based DGA detector has achieved excellent performance. However, the previous DNNs have found various inherent vulnerabilities, so cyberattackers can use these drawbacks to deceive DNNs, misleading DNNs into making wrong decisions. Backdoor attack as one of the popular attack strategies strike against DNNs has attracted widespread attention in recent years. In this paper, to cheat the LSTM-based DGA detector, we propose BadDGA, a backdoor attack against the LSTM-based DGA detector. Specifically, we offer four backdoor attack trigger construction methods: TLD-triggers, Ngram-triggers, Word-triggers, and IDN-triggers. Finally, we evaluate BadDGA on ten popular DGA datasets. The experimental results show that under the premise of 1‰ poisoning rate, our proposed backdoor attack can achieve a 100% attack success rate to verify the effectiveness of our method. Meanwhile, the model's utility on clean data is influenced slightly. [ABSTRACT FROM AUTHOR]

The Programmable Logic Controller (PLC) is located at the junction of the virtual network and physical reality in the Industrial Control System (ICS), which is vulnerable to attacks due to its weak security. Specifically, firmware tampering attacks take the firmware under the PLC operating system as the primary attack target. The firmware provides the bridge between PLC's hardware and software, which means tampering against the firmware can be more destructive and harmful than other attacks. However, existing defense and forensics methods against firmware tampering attacks are asymmetrical, which directly leads to the proliferation of such attacks and the difficulty of forensic tracing. How to accurately, quickly, and efficiently conduct forensics for such attacks is an urgent problem. In this paper, we designed and implemented a reliable detection method based on Joint Test Action Group (JTAG) and memory comparison—Aye, which can detect mainstream firmware tampering attacks reliably. To determine the effectiveness and reliability of Aye, we selected a widely used PLC to observe Aye's performance in defense and forensics by simulating the two latest PLC firmware tampering attack methods. The experimental results show that Aye can effectively defend against firmware tampering attacks, helping improve the efficiency and accuracy of such attack detection and forensics. [ABSTRACT FROM AUTHOR]

Deep learning techniques based on the neural network have made significant achievements in various fields of artificial intelligence. However, model training requires large‐scale data sets, these data sets are crowd‐sourced and model parameters will contain the encoding of private information, resulting in the risk of privacy leakage. With the trend toward sharing pretrained models, the risk of stealing training data sets through member inference attacks and model inversion attacks is further heightened. To tackle the privacy‐preserving problems in deep learning tasks, we propose an improved Differential Privacy Stochastic Gradient Descent algorithm, using Simulated Annealing algorithm and Laplace Smooth denoising mechanism to optimize the allocation method of privacy loss, replacing the constant clipping method with adaptive gradient clipping method to improve model accuracy. we also analyze privacy cost under random shuffle data batch processing method in detail within the framework of Subsampled Rényi Differential Privacy. Compared with the existing privacy protection training methods with fixed parameters and dynamic privacy parameters in classification tasks, our implementation and experiments show that we can use less privacy budget train deep neural networks with the nonconvex objective function, obtain a higher model evaluation, and have almost zero additional cost in terms of model complexity, training efficiency, and model quality. [ABSTRACT FROM AUTHOR]

The Industrial Control System (ICS) is a public facility that provides services to lots of users; thus, its security has always been a critical factor in measuring its availability. Recently, a new type of attack on ICS has occurred frequently, which realizes the extortion of users by invading the information domain and destroying the physical domain. However, due to the diversity and unavailability of an ICS control logic, the targets of such attacks are usually limited to PCs and servers, leaving more disruptive attack methods unexplored. To contribute more possible attack methods to strengthen the immunity of ICS, in this paper, we propose a novel ransomware attack method named Industrial Control System Automatic Ransomware Constructor (ICS-ARC). Compared to existing ICS ransomware, ICS-ARC can automatically generate an International Electrotechnical Commission (IEC) compliant payload to compromise the Programmable Logic Controller (PLC) without a pre-known control logic, dramatically reducing adversary requirements and leaving room for error. To evaluate the attack capability of ICS-ARC, we built a tap water treatment system as the simulation experiment target for verification. The experimental results determine that ICS-ARC can automatically generate malicious code without the control logic and complete the attack against target PLCs. In addition, to assist the related research on future attacks and defenses, we present the statistical results and corresponding analysis of PLC based on Shodan. [ABSTRACT FROM AUTHOR]

Due to the rich spatio-temporal visual content and complex multimodal relations, Video Question Answering (VideoQA) has become a challenging task and attracted increasing attention. Current methods usually leverage visual attention, linguistic attention, or self-attention to uncover latent correlations between video content and question semantics. Although these methods exploit interactive information between different modalities to improve comprehension ability, inter- and intra-modality correlations cannot be effectively integrated in a uniform model. To address this problem, we propose a novel VideoQA model called Cross-Attentional Spatio-Temporal Semantic Graph Networks (CASSG). Specifically, a multi-head multi-hop attention module with diversity and progressivity is first proposed to explore fine-grained interactions between different modalities in a crossing manner. Then, heterogeneous graphs are constructed from the cross-attended video frames, clips, and question words, in which the multi-stream spatio-temporal semantic graphs are designed to synchronously reasoning inter- and intra-modality correlations. Last, the global and local information fusion method is proposed to coalesce the local reasoning vector learned from multi-stream spatio-temporal semantic graphs and the global vector learned from another branch to infer the answer. Experimental results on three public VideoQA datasets confirm the effectiveness and superiority of our model compared with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Visual question answering (VQA) has gained increasing attention in both natural language processing and computer vision. The attention mechanism plays a crucial role in relating the question to meaningful image regions for answer inference. However, most existing VQA methods: 1) learn the attention distribution either from free-form regions or detection boxes in the image, which is intractable in answering questions about the foreground object and background form, respectively and 2) neglect the prior knowledge of human attention and learn the attention distribution with an unguided strategy. To fully exploit the advantages of attention, the learned attention distribution should focus more on the question-related image regions, such as human attention for both the questions, about the foreground object and background form. To achieve this, this article proposes a novel VQA model, called adversarial learning of supervised attentions (ALSAs). Specifically, two supervised attention modules: 1) free form-based and 2) detection-based, are designed to exploit the prior knowledge for attention distribution learning. To effectively learn the correlations between the question and image from different views, that is, free-form regions and detection boxes, an adversarial learning mechanism is implemented as an interplay between two supervised attention modules. The adversarial learning reinforces the two attention modules mutually to make the learned multiview features more effective for answer inference. The experiments performed on three commonly used VQA datasets confirm the favorable performance of ALSA. [ABSTRACT FROM AUTHOR]

These days, social media users tend to express their feelings through sharing images online. Capturing the emotions embedded in these social images involves great research challenges and practical values. Most existing works concentrate on extracting the visual feature from a global view, while ignoring the fact that visual objects are also rich in emotion. How to leverage the multilevel visual features to improve the sentiment analysis performance is important yet challenging. Besides, existing works view each social image as an independent sample while ignoring the rich correlations among social images, which may be helpful in detecting visual emotion. In this article, we propose a novel model called social relations-guided multiattention networks (SRGMANs) to incorporate both the multilevel (region-level and object-level) visual features of a single image and the correlations among multiple social images to conduct visual sentiment analysis. Specifically, we first construct a heterogeneous network consisting of various types of social relations and introduce a heterogeneous network embedding method to learn the network representation for each image. Then, two visual attention branches (region attention network and object attention network) are devised to extract emotional and discriminative visual features. For each branch, we design a self-attention module to capture the emotional dependencies among visual parts. Besides, a network-guided attention module is also designed in each branch to focus on more network-related emotional visual parts with the guidance of the topology information. Finally, the attended visual features from the two attention models, together with network representation features, are combined within a holistic framework to predict the sentiment of social images. Extensive experiments demonstrate the superiority of our model on three benchmark datasets. [ABSTRACT FROM AUTHOR]

Recent methods of extracting relational triples mainly focus on the overlapping problem and achieve considerable performance. Most previous approaches extract triples solely conditioned on context words, but ignore the potential relations among the extracted entities, which will cause incompleteness in succeeding Knowledge Graphs' (KGs) construction. Since relevant triples give a clue for establishing implicit connections among entities, we propose a Triple Relation Network (Trn) to jointly extract triples, especially handling extracting implicit triples. Specifically, we design an attention-based entity pair encoding module to identify all normal entity pairs directly. To construct implicit connections among these extracted entities in triples, we utilize our triple reasoning module to calculate relevance between two triples. Then, we select the top-K relevant triple pairs and transform them into implicit entity pairs to predict the corresponding implicit relations. We utilize a bipartite matching objective to match normal triples and implicit triples with the corresponding labels. Extensive experiments demonstrate the effectiveness of the proposed method on two public benchmarks, and our proposed model significantly outperforms previous strong baselines. [ABSTRACT FROM AUTHOR]

Due to the high exothermic characteristic of NiAl during the reaction synthesis process from Ni–Al elemental powders, the NiAl intermetallic melts frequently, and the specimens are difficult to maintain their original shape, which leads to the inhomogeneity of the pore size and morphology. To tackle this problem, porous NiAl intermetallic monolith with controllable shape and pore structure was prepared through thermal explosion (TE) using NaCl as space holder. The TE behavior was recorded, and the effect of the volume fraction of NaCl on the phase composition, macroscopic feature, pore morphology and open porosity were investigated. The results showed that NiAl was the main phase in the products, and the specimen was free from cracking or deforming when NaCl content reached 30 vol%. The interconnected channel and pore windows were formed, and the open porosity was improved greatly to 63% by adding 50 vol% NaCl. The leachable space holder route provides a simple way to control the shape, pore structure and open porosity of the synthesized porous NiAl intermetallic. The use of NaCl as space holder reduced the combustion temperature of NiAl porous intermetallic compounds and kept the macroscopic shape intact. [ABSTRACT FROM AUTHOR]

In the circumstance of social big data, sentiment analysis is attracting increasing attention for its capacity in understanding individuals’ attitudes and feelings. Traditional sentiment analysis methods focus on single modality and become ineffective as enormous data are emerging on the social websites with multiple manifestations. In this article, multimodal learning approaches are proposed to capture the relations between image and text, which only stay at the region level and ignore the fact that the channels are also closely correlated with the semantic information. In addition, social images in the social platforms are closely connected by various types of relations, which are also conducice to sentiment classification but neglected by most existing works. In this article, we propose an attention-based heterogeneous relational model to improve the multimodal sentiment analysis performance by incorporating rich social information. Specifically, we propose a progressive dual attention module to capture the correlations between image and text, and then learn the joint image-text representation from the perspective of content information. A channel attention schema is proposed here to highlight semantically rich image channels and a region attention schema is further designed to highlight the emotional regions based on the attended channels. After that, we construct a heterogeneous relation network and extend graph convolutional network to aggregate the content information from social contexts as complements to learn high-quality representations of social images. Our proposal is thoroughly evaluated on two benchmark datasets, and experimental results demonstrate the superiority of the proposed model. [ABSTRACT FROM AUTHOR]

Learning the representation for social images has recently made remarkable achievements for many tasks, such as cross-modal retrieval and multilabel classification. However, since social images contain both multimodal contents (e.g., visual images and textual descriptions) and social relations among images, simply modeling the content information may lead to suboptimal embedding. In this paper, we propose a novel multimodal representation learning model for social images, that is, correlational multimodal variational autoencoder (CMVAE) via triplet network. Specifically, in order to mine the highly nonlinear correlation between the visual content and the textual content, a CMVAE is proposed to learn a unified representation for the multiple modalities of social images. Both common information in all modalities and private information in each modality are encoded for the representation learning. To incorporate the social relations among images, we employ the triplet network to embed multiple types of social links in the representation learning. Then, a joint embedding model is proposed to combine the social relations for representation learning of the multimodal contents. Comprehensive experiment results on four datasets confirm the effectiveness of our method in two tasks, namely, multilabel classification and cross-modal retrieval. Our method outperforms the state-of-the-art multimodal representation learning methods with significant improvement of performance. [ABSTRACT FROM AUTHOR]

Porous Ni3Al, NiAl, and NiAl3 intermetallics have been synthesized through simple and rapid process of thermal explosion (TE) from 75 at.% Ni/25 at.% Al, 50 at.% Ni/50 at.% Al and 25 at.% Ni/75 at.% Al elemental powders. The effect of the three components on temperature profile, phase composition, pore structure, and oxidation resistance were investigated. The results showed that the temperature of the specimens increased rapidly from 638°C to 1381°C, 620°C to 1613°C and 642°C to 1172°C in a few seconds. X-ray diffraction patterns indicated that Ni3Al, NiAl, and NiAl3 were the main phase or single-phase compounds in the TE products. Porous Ni3Al, NiAl, and NiAl3 intermetallics showed variable pore structures. Moreover, all the products of three components exhibited good oxidation resistance, particularly the porous NiAl and NiAl3 intermetallics. [ABSTRACT FROM AUTHOR]

Porous Ni3Al, NiAl, and NiAl3 intermetallics have been synthesized through simple and rapid process of thermal explosion (TE) from 75 at.% Ni/25 at.% Al, 50 at.% Ni/50 at.% Al and 25 at.% Ni/75 at.% Al elemental powders. The effect of the three components on temperature profile, phase composition, pore structure, and oxidation resistance were investigated. The results showed that the temperature of the specimens increased rapidly from 638°C to 1381°C, 620°C to 1613°C and 642°C to 1172°C in a few seconds. X-ray diffraction patterns indicated that Ni3Al, NiAl, and NiAl3 were the main phase or single-phase compounds in the TE products. Porous Ni3Al, NiAl, and NiAl3 intermetallics showed variable pore structures. Moreover, all the products of three components exhibited good oxidation resistance, particularly the porous NiAl and NiAl3 intermetallics. [ABSTRACT FROM AUTHOR]

We study the problem of response selection for multi-turn conversation in retrieval-based chatbots. The task involves matching a response candidate with a conversation context, the challenges for which include how to recognize important parts of the context, and how to model the relationships among utterances in the context. Existing matching methods may lose important information in contexts as we can interpret them with a unified framework in which contexts are transformed to fixed-length vectors without any interaction with responses before matching. This motivates us to propose a new matching framework that can sufficiently carry important information in contexts to matching and model relationships among utterances at the same time. The new framework, which we call a sequential matching framework (SMF), lets each utterance in a context interact with a response candidate at the first step and transforms the pair to a matching vector. The matching vectors are then accumulated following the order of the utterances in the context with a recurrent neural network (RNN) that models relationships among utterances. Context-response matching is then calculated with the hidden states of the RNN. Under SMF, we propose a sequential convolutional network and sequential attention network and conduct experiments on two public data sets to test their performance. Experiment results show that both models can significantly outperform state-of-the-art matching methods. We also show that the models are interpretable with visualizations that provide us insights on how they capture and leverage important information in contexts for matching. [ABSTRACT FROM AUTHOR]

Landmark retrieval is to return a set of images with their landmarks similar to those of the query images. Existing studies on landmark retrieval focus on exploiting the geometries of landmarks for visual similarity matches. However, the visual content of social images is of large diversity in many landmarks, and also some images share common patterns over different landmarks. On the other side, it has been observed that social images usually contain multimodal contents, i.e., visual content and text tags, and each landmark has the unique characteristic of both visual content and text content. Therefore, the approaches based on similarity matching may not be effective in this environment. In this paper, we investigate whether the geographical correlation among the visual content and the text content could be exploited for landmark retrieval. In particular, we propose an effective multimodal landmark classification paradigm to leverage the multimodal contents of social image for landmark retrieval, which integrates feature refinement and landmark classifier with multimodal contents by a joint model. The geo-tagged images are automatically labeled for classifier learning. Visual features are refined based on low rank matrix recovery, and multimodal classification combined with group sparse is learned from the automatically labeled images. Finally, candidate images are ranked by combining classification result and semantic consistence measuring between the visual content and text content. Experiments on real-world datasets demonstrate the superiority of the proposed approach as compared to existing methods. [ABSTRACT FROM PUBLISHER]

Previous work on the one-class collaborative filtering (OCCF) problem can be roughly categorized into pointwise methods, pairwise methods, and content-based methods. A fundamental assumption of these approaches is that all missing values in the user-item rating matrix are considered negative. However, this assumption may not hold because the missing values may contain negative and positive examples. For example, a user who fails to give positive feedback about an item may not necessarily dislike it; he may simply be unfamiliar with it. Meanwhile, content-based methods, e.g. collaborative topic regression (CTR), usually require textual content information of the items, and thus their applicability is largely limited when the text information is not available. In this paper, we propose to apply the latent Dirichlet allocation (LDA) model on OCCF to address the above-mentioned problems. The basic idea of this approach is that items are regarded as words, users are considered as documents, and the user-item feedback matrix constitutes the corpus. Our model drops the strong assumption that missing values are all negative and only utilizes the observed data to predict a user’s interest. Additionally, the proposed model does not need content information of the items. Experimental results indicate that the proposed method outperforms previous methods on various ranking-oriented evaluation metrics. We further combine this method with a matrix factorization-based method to tackle the multi-class collaborative filtering (MCCF) problem, which also achieves better performance on predicting user ratings. [ABSTRACT FROM AUTHOR]

In the design of differentially private mechanisms, it is usually assumed that a uniformly random source is available. However, in many situations it seems unrealistic, and one must deal with various imperfect random sources. Dodis et al. (CRYPTO’12) presented that differential privacy can be achieved with Santha–Vazirani (SV) sources, introduced in 1986, via adding a stronger property called SV-consistent sampling and left open the question if differential privacy is possible with more realistic, i.e., less structured, sources. A new source, called bias-control limited (BCL) source, introduced by Dodis (ICALP’01), is more realistic. It can be considered as a generalization of the SV and sequential bit-fixing sources. Unfortunately, the natural extension of SV-consistent sampling to the BCL source is hopeless to achieve differential privacy, mainly because SV-consistent sampling requires “consecutive” strings, while some strings cannot be generated from “non-trivial” BCL sources. Moreover, there exists some inconsistency in the bit lengths of random sources between the mechanism construction and the proofs for differentially privacy presented by Dodis et al. (CRYPTO’12). More appropriate mechanisms and more rigourous proofs are needed. Motivated by these questions, we introduce a new appealing property, called compact BCL-consistent sampling, the degeneration of which is different from SV-consistent sampling shown by Dodis et al. (CRYPTO’12). We prove that if the mechanism based on the BCL source satisfies this property, then it is differentially private for certain parameters. Even if the BCL source is degenerated into the SV source, our proof is much more intuitive and simpler than that of Dodis et al. Further, we construct an explicit finite-precision mechanism using an improved construction of infinite mechanism and a new truncation technique. We also propose its concrete results for differential privacy and utility, one core proof step of which is trickier and much shorter than the counterpart by Dodis et al. (CRYPTO’12). While the results of Dodis and Yao (CRYPTO’15) imply that if there exist differentially private mechanisms for imperfect randomness, then the parameters should have some constraints, we show an explicit construction of such mechanisms, whose parameters match the prior constraints. [ABSTRACT FROM PUBLISHER]

There is a growing volume of social images with the development of social networks and digital cameras. Usually, these images are annotated with textual tags besides the visual content. It is quite urgent to automatically organize and manage this large number of social images. Image classification is the basic task of these applications and has attracted great research efforts. Though there are many researches on image classification, it is of considerable challenge to integrate the multi-modal content of social images simultaneously for classification, since the textual content and visual content are represented in two heterogeneous feature spaces. In this paper, we proposed a multi-modal learning method to integrate multi-modal features through their correlation seamlessly. Specifically, we learn two linear classification modules for the two types of features, and then they are integrated by the l2 normalization method via a joint model. Each classier is normalized with l2,1 to reduce the effect of the noisy features by selecting a subset of more important features. With the joint model, the classification based on visual features can be reinforced by the classification based on textual features, and vice verse. Then, the test image is classified based on both the textual features and visual features by combing the results of the two classifiers. Experiments conducted on real-world social image datasets demonstrate the superiority of our proposed method compared with the representative baselines. [ABSTRACT FROM AUTHOR]

Studying the bursty nature of cascades in social media is practically important in many real applications such as product sales prediction, disaster relief, and stock market prediction. Although both the cascade size prediction and the burst patterns of the cascades have been extensively studied, how to predict when a burst will come remains an open problem. It is challenging for traditional time-series-based models such as regression models to address this task directly. Firstly, times-series-based prediction models focus on predicting the future values based on previously observed ones. It is hard to apply them to predict the time of a bursts with the 'quick rise-and-fall' pattern. Secondly, besides the cascade popularity, a lot of other side information like user profile and social relation are available in social media. Although the potential utility of such information can be high, it is also hard for time-series-based models to capture and integrate these rich information with diverse formats seamlessly. This paper proposes a classification-based approach for burst time prediction by exploiting rich knowledge in information diffusion. Particularly, we first propose a time-window-based transformation to predict in which time window the burst will appear. By dividing the time spans of all the cascades into the same number of time windows K, the cascades with diverse time spans can thus be handled uniformly. To exploit the rich and heterogenous information in social media, we next propose a scale-independent feature extraction framework to model the heterogenous knowledge in a scale-independent manner. Systematical evaluations are conducted on the Sina Weibo reposting dataset and MemeTracker dataset. Besides the superior performance of the proposed approach, we also observe that: (1) surprisingly, social/structure knowledge is more indicative of the bursts than the cascade popularity information, especially for the bursts occurring in a farther future. (2) Larger cascades are harder to predict as the spreading process of the cascades with higher popularity is usually more diverse and fluctuant. (3) The proposed approach is robust in the sense that the result is not much sensitive to the popularity of the training cascades. [ABSTRACT FROM AUTHOR]

Image location prediction is used to estimate the geolocation where an image is taken, which is important for many image applications, such as image retrieval, image browsing, and organization. Since a social image contains heterogeneous contents, such as visual content and textual content, effectively incorporating these contents to predict location is nontrivial. Moreover, it is observed that image content patterns and the locations where they may appear correlate hierarchically. Traditional image location prediction methods mainly adopt a single-level architecture and assume images are independently distributed in geographical space, which is not directly adaptable to the hierarchical correlation. In this paper, we propose a geographically hierarchical bi-modal deep belief network (GH-BDBN) model, which is a compositional learning architecture that integrates multi-modal deep learning model with a non-parametric hierarchical prior model. GH-BDBN learns a joint representation capturing the correlations among different types of image content using a bi-modal DBN, with a geographically hierarchical prior over the joint representation to model the hierarchical correlation between image content and location. Then, an efficient inference algorithm is proposed to learn the parameters and the geographical hierarchical structure of geographical locations. Experimental results demonstrate the superiority of our model for image location prediction. [ABSTRACT FROM PUBLISHER]

In a real e-commerce website, usually only a small number of users will give ratings to the items they purchased, and this can lead to the very sparse user-item rating data. The data sparsity issue will greatly limit the recommendation performance of most recommendation algorithms. However, a user may register accounts in many e-commerce websites. If such users’ historical purchasing data on these websites can be integrated, the recommendation performance could be improved. But it is difficult to align the users and items between these websites, and thus how to effectively borrow the users’ rating data of one website (source domain) to help improve the recommendation performance of another website (target domain) is very challenging. To this end, this paper extended the traditional one-dimensional psychometrics model to multidimension. The extended model can effectively capture users’ multiple interests. Based on this multidimensional psychometrics model, we further propose a novel transfer learning algorithm. It can effectively transfer users’ rating preferences from the source domain to the target domain. Experimental results show that the proposed method can significantly improve the recommendation performance. [ABSTRACT FROM AUTHOR]

With the rapidly increasing popularity of social media sites, a large amount of user-generated data has been injected into the web. The data include a wide variety of real-world events. As a consequence, especially for social multimedia objects, it has become increasingly difficult to allow the browsing and organization of multimedia collections in a more effective manner. The approach we propose in this study addresses this problem, thus enabling the browsing and organization of multimedia collections in a natural way, i.e., by events. There have been some research studies on this problem. However, most of the previous approaches merge multiple types of features (e.g., textual content, visual content, user information and temporal information) of social media using a relatively simple mechanism. In this study, we merge multiple types of features in an integrated manner to identify the event associated with user-contributed social multimedia objects. We exploit the correlations between different types of features, i.e., textual content, visual content, user information and temporal information, to classify new social multimedia objects into their corresponding event categories. We accomplish this through a feature correlation graph ( FCG) that uses features as nodes and the correlations among these features as edges for each event and individual multimedia object. We then employ a probabilistic model based on Markov random field to connect each new multimedia object with the correct event. We evaluate the algorithm on large-scale, real-world datasets of event images downloaded from Flickr, and the experimental results confirm the superiority of our approach over state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

With the availability of cheap location sensors, geotagging of images in online social media is very popular. With a large amount of geo-tagged social images, it is interesting to study how these images are shared across geographical regions and how the geographical language characteristics and vision patterns are distributed across different regions. Unlike textual document, geo-tagged social image contains multiple types of content, i.e., textual description, visual content, and geographical information. Existing approaches usually mine geographical characteristics using a subset of multiple types of image contents or combining those contents linearly, which ignore correlations between different types of contents, and their geographical distributions. Therefore, in this paper, we propose a novel method to discover geographical characteristics of geo-tagged social images using a geographical topic model called geographical topic model of social images (GTMSIs). GTMSI integrates multiple types of social image contents as well as the geographical distributions, in which image topics are modeled based on both vocabulary and visual features. In GTMSI, each region of the image would have its own topic distribution, and hence have its own language model and vision pattern. Experimental results show that our GTMSI could identify interesting topics and vision patterns, as well as provide location prediction and image tagging. [ABSTRACT FROM PUBLISHER]