Search

This paper presents an ontology design along with knowledge engineering, and multilingual semantic reasoning techniques to build an automated system for assimilating culinary information for Indian food in the form of a knowledge graph. The main focus is on designing intelligent methods to derive ontology designs and capture all-encompassing knowledge about food, recipes, ingredients, cooking characteristics, and most importantly, nutrition, at scale. We present our ongoing work in this workshop paper, describe in some detail the relevant challenges in curating knowledge of Indian food, and propose our high-level ontology design. We also present a novel workflow that uses AI, LLM, and language technology to curate information from recipe blog sites in the public domain to build knowledge graphs for Indian food. The methods for knowledge curation proposed in this paper are generic and can be replicated for any domain. The design is application-agnostic and can be used for AI-driven smart analysis, building recommendation systems for Personalized Digital Health, and complementing the knowledge graph for Indian food with contextual information such as user information, food biochemistry, geographic information, agricultural information, etc., Comment: 14 pages, 3 figures, 25 references, Formal Ontology in Information Systems Conference 2024 - Integrated Food Ontology Workshop

Summarizing software artifacts is an important task that has been thoroughly researched. For evaluating software summarization approaches, human judgment is still the most trusted evaluation. However, it is time-consuming and fatiguing for evaluators, making it challenging to scale and reproduce. Large Language Models (LLMs) have demonstrated remarkable capabilities in various software engineering tasks, motivating us to explore their potential as automatic evaluators for approaches that aim to summarize software artifacts. In this study, we investigate whether LLMs can evaluate bug report summarization effectively. We conducted an experiment in which we presented the same set of bug summarization problems to humans and three LLMs (GPT-4o, LLaMA-3, and Gemini) for evaluation on two tasks: selecting the correct bug report title and bug report summary from a set of options. Our results show that LLMs performed generally well in evaluating bug report summaries, with GPT-4o outperforming the other LLMs. Additionally, both humans and LLMs showed consistent decision-making, but humans experienced fatigue, impacting their accuracy over time. Our results indicate that LLMs demonstrate potential for being considered as automated evaluators for bug report summarization, which could allow scaling up evaluations while reducing human evaluators effort and fatigue.

Digital libraries often face the challenge of processing a large volume of diverse document types. The manual collection and tagging of metadata can be a time-consuming and error-prone task. To address this, we aim to develop an automatic metadata extractor for digital libraries. In this work, we introduce the Heterogeneous Learning Resources (HLR) dataset designed for document image classification. The approach involves decomposing individual learning resources into constituent document images (sheets). These images are then processed through an OCR tool to extract textual representation. State-of-the-art classifiers are employed to classify both the document image and its textual content. Subsequently, the labels of the constituent document images are utilized to predict the label of the overall document., Comment: 2 pages, 3 figures, Published in JCDL 21

The Information Retrieval in Software Engineering (IRSE) track aims to develop solutions for automated evaluation of code comments in a machine learning framework based on human and large language model generated labels. In this track, there is a binary classification task to classify comments as useful and not useful. The dataset consists of 9048 code comments and surrounding code snippet pairs extracted from open source github C based projects and an additional dataset generated individually by teams using large language models. Overall 56 experiments have been submitted by 17 teams from various universities and software companies. The submissions have been evaluated quantitatively using the F1-Score and qualitatively based on the type of features developed, the supervised learning model used and their corresponding hyper-parameters. The labels generated from large language models increase the bias in the prediction model but lead to less over-fitted results., Comment: Overview Paper of the Information Retrieval of Software Engineering Track at the Forum for Information Retrieval, 2023

Syntactic foams are a promising candidate for applications in marine and oil and gas industries in underwater cables and pipelines due to their excellent insulation properties. The effective transmission of electrical energy through cables requires insulation materials with a low loss factor and low dielectric constant. Similarly, in transporting fluid through pipelines, thermal insulation is crucial. However, both applications are susceptible to potential environmental degradation from moisture exposure, which can significantly impact the material's properties. This study addresses the knowledge gap by examining the implications of prolonged moisture exposure on TPU and TPU-derived syntactic foam via various multi-scale materials characterization methods. The research focuses on a flexible syntactic foam created using selective laser sintering and thermoplastic polyurethane elastomer (TPU) reinforced with glass microballoons (GMB). The study specifically explores the impact of moisture exposure duration and GMB volume fraction on microphase morphological changes, their associated mechanisms, and their influence on thermal transport and dielectric properties., Comment: arXiv admin note: substantial text overlap with arXiv:2307.13673

To address the issue of rising software maintenance cost due to program comprehension challenges, we propose SMARTKT (Smart Knowledge Transfer), a search framework, which extracts and integrates knowledge related to various aspects of an application in form of a semantic graph. This graph supports syntax and semantic queries and converts the process of program comprehension into a {\em google-like} search problem., Comment: 3 pages, 2 figures, accepted in the NDLI-UNESCO International Symposium on Knowledge Engineering for Digital Library Design 2019 (KEDL) as an extended abstract and poster

Topic modeling has emerged as a dominant method for exploring large document collections. Recent approaches to topic modeling use large contextualized language models and variational autoencoders. In this paper, we propose a negative sampling mechanism for a contextualized topic model to improve the quality of the generated topics. In particular, during model training, we perturb the generated document-topic vector and use a triplet loss to encourage the document reconstructed from the correct document-topic vector to be similar to the input document and dissimilar to the document reconstructed from the perturbed vector. Experiments for different topic counts on three publicly available benchmark datasets show that in most cases, our approach leads to an increase in topic coherence over that of the baselines. Our model also achieves very high topic diversity., Comment: Accepted at 19th International Conference on Natural Language Processing (ICON 2022)

Nowadays many research articles are prefaced with research highlights to summarize the main findings of the paper. Highlights not only help researchers precisely and quickly identify the contributions of a paper, they also enhance the discoverability of the article via search engines. We aim to automatically construct research highlights given certain segments of a research paper. We use a pointer-generator network with coverage mechanism and a contextual embedding layer at the input that encodes the input tokens into SciBERT embeddings. We test our model on a benchmark dataset, CSPubSum, and also present MixSub, a new multi-disciplinary corpus of papers for automatic research highlight generation. For both CSPubSum and MixSub, we have observed that the proposed model achieves the best performance compared to related variants and other models proposed in the literature. On the CSPubSum dataset, our model achieves the best performance when the input is only the abstract of a paper as opposed to other segments of the paper. It produces ROUGE-1, ROUGE-2 and ROUGE-L F1-scores of 38.26, 14.26 and 35.51, respectively, METEOR score of 32.62, and BERTScore F1 of 86.65 which outperform all other baselines. On the new MixSub dataset, where only the abstract is the input, our proposed model (when trained on the whole training corpus without distinguishing between the subject categories) achieves ROUGE-1, ROUGE-2 and ROUGE-L F1-scores of 31.78, 9.76 and 29.3, respectively, METEOR score of 24.00, and BERTScore F1 of 85.25., Comment: 19 Pages, 7 Figures, 8 Tables

Melody extraction is a vital music information retrieval task among music researchers for its potential applications in education pedagogy and the music industry. Melody extraction is a notoriously challenging task due to the presence of background instruments. Also, often melodic source exhibits similar characteristics to that of the other instruments. The interfering background accompaniment with the vocals makes extracting the melody from the mixture signal much more challenging. Until recently, classical signal processing-based melody extraction methods were quite popular among melody extraction researchers. The ability of the deep learning models to model large-scale data and the ability of the models to learn automatic features by exploiting spatial and temporal dependencies inspired many researchers to adopt deep learning models for melody extraction. In this paper, an attempt has been made to review the up-to-date data-driven deep learning approaches for melody extraction from polyphonic music. The available deep models have been categorized based on the type of neural network used and the output representation they use for predicting melody. Further, the architectures of the 25 melody extraction models are briefly presented. The loss functions used to optimize the model parameters of the melody extraction models are broadly categorized into four categories and briefly describe the loss functions used by various melody extraction models. Also, the various input representations adopted by the melody extraction models and the parameter settings are deeply described. A section describing the explainability of the block-box melody extraction deep neural networks is included. The performance of 25 melody extraction methods is compared. The possible future directions to explore/improve the melody extraction methods are also presented in the paper., Comment: 72 pages

Here we explore the use of scanning electron diffraction coupled with electron atomic pair distribution function analysis (ePDF) to understand the local order as a function of position in a complex multicomponent system, a hot rolled, Ni-encapsulated, Zr$_{65}$Cu$_{17.5}$Ni$_{10}$Al$_{7.5}$ bulk metallic glass (BMG), with a spatial resolution of 3 nm. We show that it is possible to gain insight into the chemistry and chemical clustering/ordering tendency in different regions of the sample, including in the vicinity of nano-scale crystallites that are identified from virtual dark field images and in heavily deformed regions at the edge of the BMG. In addition to simpler analysis, unsupervised machine learning was used to extract partial PDFs from the material, modeled as a quasi-binary alloy, and map them in space. These maps allowed key insights not only into the local average composition, as validated by EELS, but also a unique insight into chemical short-range ordering tendencies in different regions of the sample during formation. The experiments are straightforward and rapid and, unlike spectroscopic measurements, don't require energy filters on the instrument. We spatially map different quantities of interest (QoI's), defined as scalars that can be computed directly from positions and widths of ePDF peaks or parameters refined from fits to the patterns. We developed a flexible and rapid data reduction and analysis software framework that allows experimenters to rapidly explore images of the sample on the basis of different QoI's. The power and flexibility of this approach are explored and described in detail. Because of the fact that we are getting spatially resolved images of the nanoscale structure obtained from ePDFs we call this approach scanning nano-structure electron microscopy (SNEM), and we believe that it will be powerful and useful extension of current 4D-STEM methods.

Topical Segmentation poses a great role in reducing search space of the topics taught in a lecture video specially when the video metadata lacks topic wise segmentation information. This segmentation information eases user efforts of searching, locating and browsing a topic inside a lecture video. In this work we propose an algorithm, that combines state-of-the art language model and domain knowledge graph for automatically detecting different coherent topics present inside a long lecture video. We use the language model on speech-to-text transcription to capture the implicit meaning of the whole video while the knowledge graph provides us the domain specific dependencies between different concepts of that subjects. Also leveraging the domain knowledge we can capture the way instructor binds and connects different concepts while teaching, which helps us in achieving better segmentation accuracy. We tested our approach on NPTEL lecture videos and holistic evaluation shows that it out performs the other methods described in the literature.

Singing Voice Detection (SVD) has been an active area of research in music information retrieval (MIR). Currently, two deep neural network-based methods, one based on CNN and the other on RNN, exist in literature that learn optimized features for the voice detection (VD) task and achieve state-of-the-art performance on common datasets. Both these models have a huge number of parameters (1.4M for CNN and 65.7K for RNN) and hence not suitable for deployment on devices like smartphones or embedded sensors with limited capacity in terms of memory and computation power. The most popular method to address this issue is known as knowledge distillation in deep learning literature (in addition to model compression) where a large pre-trained network known as the teacher is used to train a smaller student network. Given the wide applications of SVD in music information retrieval, to the best of our knowledge, model compression for practical deployment has not yet been explored. In this paper, efforts have been made to investigate this issue using both conventional as well as ensemble knowledge distillation techniques., Comment: Accepted at INTERSPEECH 2021. 5 pages, 3 figures

Bharatanatyam, an Indian Classical Dance form, represents the rich cultural heritage of India. Analysis and recognition of such dance forms are critical for the preservation of cultural heritage. Like in most dance forms, a Bharatanatyam dancer performs in synchronization with structured rhythmic music, called Sollukattu, which comprises instrumental beats and vocalized utterances (bols) to create a rhythmic music structure. Computer analysis of Bharatanatyam, therefore, requires a structural analysis of Sollukattus. In this paper, we use speech processing techniques to recognize bols. Exploiting the predefined structures of Sollukattus and the detected bols, we recognize the Sollukattu. We estimate the tempo period by two methods. Finally, we generate a complete annotation of the audio signal by beat marking. For this, we also use the information of beats detected from the onset envelope of a Sollukattu signal. For training and test, we create a data set for Sollukattus and annotate them. We achieve 85% accuracy in bol recognition, 95% in Sollukattu recognition, 96% in tempo period estimation, and over 90% in beat marking. This is the maiden attempt to fully structurally analyze the music of an Indian Classical Dance form and the use of speech processing techniques for beat marking.

We report a model to predict patient's radiological response to curative radiation therapy (RT) for non-small-cell lung cancer (NSCLC). Cone-Beam Computed Tomography images acquired weekly during the six-week course of RT were contoured with the Gross Tumor Volume (GTV) by senior radiation oncologists for 53 patients (7 images per patient). Deformable registration of the images yielded six deformation fields for each pair of consecutive images per patient. Jacobian of a field provides a measure of local expansion/contraction and is used in our model. Delineations were compared post-registration to compute unchanged ($U$), newly grown ($G$), and reduced ($R$) regions within GTV. The mean Jacobian of these regions $\mu_U$, $\mu_G$ and $\mu_R$ are statistically compared and a response assessment model is proposed. A good response is hypothesized if $\mu_R < 1.0$, $\mu_R < \mu_U$, and $\mu_G < \mu_U$. For early prediction of post-treatment response, first, three weeks' images are used. Our model predicted clinical response with a precision of $74\%$. Using reduction in CT numbers (CTN) and percentage GTV reduction as features in logistic regression, yielded an area-under-curve of 0.65 with p=0.005. Combining logistic regression model with the proposed hypothesis yielded an odds ratio of 20.0 (p=0.0).

Prediction of survivability in a patient for tumor progression is useful to estimate the effectiveness of a treatment protocol. In our work, we present a model to take into account the heterogeneous nature of a tumor to predict survival. The tumor heterogeneity is measured in terms of its mass by combining information regarding the radiodensity obtained in images with the gross tumor volume (GTV). We propose a novel feature called Tumor Mass within a GTV (TMG), that improves the prediction of survivability, compared to existing models which use GTV. Weekly variation in TMG of a patient is computed from the image data and also estimated from a cell survivability model. The parameters obtained from the cell survivability model are indicatives of changes in TMG over the treatment period. We use these parameters along with other patient metadata to perform survival analysis and regression. Cox's Proportional Hazard survival regression was performed using these data. Significant improvement in the average concordance index from 0.47 to 0.64 was observed when TMG is used in the model instead of GTV. The experiments show that there is a difference in the treatment response in responsive and non-responsive patients and that the proposed method can be used to predict patient survivability., Comment: Under review in SPIE Journal of Medical Imaging

Understanding the underlying semantics of performing arts like dance is a challenging task. Dance is multimedia in nature and spans over time as well as space. Capturing and analyzing the multimedia content of the dance is useful for the preservation of cultural heritage, to build video recommendation systems, to assist learners to use tutoring systems. To develop an application for dance, three aspects of dance analysis need to be addressed: 1) Segmentation of the dance video to find the representative action elements, 2) Matching or recognition of the detected action elements, and 3) Recognition of the dance sequences formed by combining a number of action elements under certain rules. This paper attempts to solve three fundamental problems of dance analysis for understanding the underlying semantics of dance forms. Our focus is on an Indian Classical Dance (ICD) form known as Bharatanatyam. As dance is driven by music, we use the music as well as motion information for key posture extraction. Next, we recognize the key postures using machine learning as well as deep learning techniques. Finally, the dance sequence is recognized using the Hidden Markov Model (HMM). We capture the multi-modal data of Bharatanatyam dance using Kinect and build an annotated data set for research in ICD.

Conventional Convolutional neural networks (CNN) are trained on large domain datasets and are hence typically over-represented and inefficient in limited class applications. An efficient way to convert such large many-class pre-trained networks into small few-class networks is through a hierarchical decomposition of its feature maps. To alleviate this issue, we propose an automated framework for such decomposition in Hierarchically Self Decomposing CNN (HSD-CNN), in four steps. HSD-CNN is derived automatically using a class-specific filter sensitivity analysis that quantifies the impact of specific features on a class prediction. The decomposed hierarchical network can be utilized and deployed directly to obtain sub-networks for a subset of classes, and it is shown to perform better without the requirement of retraining these sub-networks. Experimental results show that HSD-CNN generally does not degrade accuracy if the full set of classes are used. Interestingly, when operating on known subsets of classes, HSD-CNN has an improvement in accuracy with a much smaller model size, requiring much fewer operations. HSD-CNN flow is verified on the CIFAR10, CIFAR100 and CALTECH101 data sets. We report accuracies up to $85.6\%$ ( $94.75\%$ ) on scenarios with 13 ( 4 ) classes of CIFAR100, using a pre-trained VGG-16 network on the full data set. In this case, the proposed HSD-CNN requires $3.97 \times$ fewer parameters and has $71.22\%$ savings in operations, in comparison to baseline VGG-16 containing features for all 100 classes., Comment: Accepted in ICVGIP,2018

Deformable registration of spatiotemporal Cone-Beam Computed Tomography (CBCT) images taken sequentially during the radiation treatment course yields a deformation field for a pair of images. The Jacobian of this field at any voxel provides a measure of the expansion or contraction of a unit volume. We analyze the Jacobian at different sections of the tumor volumes obtained from delineation done by radiation oncologists for lung cancer patients. The delineations across the temporal sequence are compared post registration to compute tumor areas namely, unchanged (U), newly grown (G), and reduced (R) that have undergone changes. These three regions of the tumor are considered for statistical analysis. In addition, statistics of non-tumor (N) regions are taken into consideration. Sequential CBCT images of 29 patients were used in studying the distribution of Jacobian in these four different regions, along with a test set of 16 patients. Statistical tests performed over the dataset consisting of first three weeks of treatment suggest that, means of the Jacobian in the regions follow a particular order. Although, this observation is apparent when applied to the distribution over the whole population, it is found that the ordering deviates for many individual cases. We propose a hypothesis to classify patients who have had partial response (PR). Early prediction of the response was studied using only three weeks of data. The early prediction of response of treatment was supported by a Fisher's test with odds ratio of 5.13 and a p-value of 0.043.