Search

We address personalized image captioning, which generates a descriptive sentence for a user's image, accounting for prior knowledge such as her active vocabulary or writing style in her previous documents. As applications of personalized image captioning, we solve two post automation tasks in social networks: hashtag prediction and post generation. The hashtag prediction predicts a list of hashtags for an image, while the post generation creates a natural text consisting of normal words, emojis, and even hashtags. We propose a novel personalized captioning model named Context Sequence Memory Network (CSMN). Its unique updates over existing memory networks include (i) exploiting memory as a repository for multiple types of context information, (ii) appending previously generated words into memory to capture long-term information, and (iii) adopting CNN memory structure to jointly represent nearby ordered memory slots for better context understanding. For evaluation, we collect a new dataset InstaPIC-1.1M, comprising 1.1M Instagram posts from 6.3 K users. We further use the benchmark YFCC100M dataset  to validate the generality of our approach. With quantitative evaluation and user studies via Amazon Mechanical Turk, we show that the three novel features of the CSMN help enhance the performance of personalized image captioning over state-of-the-art captioning models. [ABSTRACT FROM AUTHOR]

We propose an approach for retrieving a sequence of natural sentences for an image stream. Since general users often take a series of pictures on their experiences, much online visual information exists in the form of image streams, for which it would better take into consideration of the whole image stream to produce natural language descriptions. While almost all previous studies have dealt with the relation between a single image and a single natural sentence, our work extends both input and output dimension to a sequence of images and a sequence of sentences. For retrieving a coherent flow of multiple sentences for a photo stream, we propose a multimodal neural architecture called coherence recurrent convolutional network (CRCN), which consists of convolutional neural networks, bidirectional long short-term memory (LSTM) networks, and an entity-based local coherence model. Our approach directly learns from vast user-generated resource of blog posts as text-image parallel training data. We collect more than 22 K unique blog posts with 170 K associated images for the travel topics of NYC, Disneyland , Australia, and Hawaii. We demonstrate that our approach outperforms other state-of-the-art image captioning methods for text sequence generation, using both quantitative measures and user studies via Amazon Mechanical Turk. [ABSTRACT FROM PUBLISHER]

MPEG-4 video streaming service is being regarded as a de facto standard service for current mobile multimedia video streaming services such as video conferencing, voice over IP, and even state-of-the-art digital multimedia broadcasting technology. There have been many illegal users who have used copyrighted material without getting permission and/or paying for its use. Therefore, much attention has been paid for Digital Rights Management (DRM) for digital media. We constructed a protection scheme for MPEG-4 video file format, which can be applied, to the video services that are exploiting the MPEG-4 standard In this protection scheme, minimum segments of every Video Object Plane in a MPEG-4 video file could be encrypted with a symmetric cryptography system like a DES (Data Encryption Standard) so that people who have not received permission and/or paid to use the contents would not be able to view them. We applied this scheme to all kinds of MPEG-4 VOP (Video Object Plane) types, that is I-, P-, and B-VOPs respectively. The prototype system was developed for proofing the feasibility of this scheme for hand-held devices and for mobile multimedia service environment. The efficiency of this scheme is reported at the end of this paper. [ABSTRACT FROM AUTHOR]