Search

Object detection in embedded systems is important for many contemporary applications that involve vision and scene analysis. In this paper, we propose a novel architecture for object detection implemented in FPGA, based on the Stripe Memory Engine (SME), and point out shortcomings of existing architectures. SME processes a stream of image data so that it stores a narrow stripe of the input image and its scaled versions and uses a detector unit which is efficiently pipelined across multiple image positions within the SME. We show how to process images with up to 4K resolution at high frame rates using cascades of SMEs. As a detector algorithm, the SMEs use boosted soft cascade with simple image features that require only pixel comparisons and look-up tables; therefore, they are well suitable for hardware implemenation. We describe the components of our architecture and compare it to several published works in several configurations. As an example, we implemented face detection and license plate detection applications that work with HD images ($1280\times 720$ pixels) running at over 60 frames/s on Xilinx Zynq platform. We analyzed their power consumption, evaluated the accuracy of our detectors, and compared them to Haar Cascades from OpenCV that are often used by other authors. We show that our detectors offer better accuracy as well as performance at lower power consumption. [ABSTRACT FROM AUTHOR]

According to a recent Deloitte study, the COVID-19 pandemic continues to place a huge strain on the global health care sector. Covid-19 has also catalysed digital transformation across the sector for improving operational efficiencies. As a result, the amount of digitally stored patient data such as discharge letters, scan images, test results or free text entries by doctors has grown significantly. In 2020, 2314 exabytes of medical data was generated globally. This medical data does not conform to a generic structure and is mostly in the form of unstructured digitally generated or scanned paper documents stored as part of a patient’s medical reports. This unstructured data is digitised using Optical Character Recognition (OCR) process. A key challenge here is that the accuracy of the OCR process varies due to the inability of current OCR engines to correctly transcribe scanned or handwritten documents in which text may be skewed, obscured or illegible. This is compounded by the fact that processed text is comprised of specific medical terminologies that do not necessarily form part of general language lexicons. The proposed work uses a deep neural network based self-supervised pre-training technique: Robustly Optimized Bidirectional Encoder Representations from Transformers (RoBERTa) that can learn to predict hidden (masked) sections of text to fill in the gaps of non-transcribable parts of the documents being processed. Evaluating the proposed method on domain-specific datasets which include real medical documents, shows a significantly reduced word error rate demonstrating the effectiveness of the approach. [ABSTRACT FROM AUTHOR]

Belief propagation (BP)-based stereo matching has popular owing to its regularity and ability to yield promising results. Some commonly observed hardware-implementation challenges pertaining to the use of this algorithm are large memory requirements and trade-offs between speed and chip area, along with an increasing disparity range. The paper presents a hardware- and memory-efficient architecture for building a BP-based disparity estimation system capable of overcoming issues associated with large disparity ranges. The proposed architecture is memory-efficient owing to the regularity of its underlying algorithm. In addition, the improved hardware efficiency can be attributed to processing element modifications to demonstrate shareable characteristics. Results obtained in this study reveal a 67.8% reduction in required memory corresponding to a time–area term complexity of $O(L(logL)^{2})$ , where L denotes the disparity range. This result is in stark contrast to the $O(L^{2}logL)$ and $O(L^{2})$ complexities observed in extant studies. Compared to state-of-the-art implementations, the proposed architecture offers an 86.2% gate count reduction for message update units at a disparity range of 512. These results confirm the proposed architecture’s suitability for use in large disparity scenarios. [ABSTRACT FROM AUTHOR]

Guided image filtering (GIF) is a promising edge-preserving filtering technique that has been applied in a variety of applications. Nevertheless, an efficient very-large-scale integration (VLSI) architecture design of GIF is still very challenging for the real-time processing of full-high definition videos. Previously proposed architectures are somewhat inefficient in terms of either on-chip memory usage or off-chip memory bandwidth. This paper aims to improve the balance between on-chip memory usage and off-chip memory bandwidth through architecture exploration. Three critical architectural tradeoffs in the VLSI design of GIF are explored, and two efficient VLSI architectures, namely sequential line-based and parallel line-based architectures, are proposed. Experimental results demonstrate that the proposed VLSI design only consumes 34.1-K logic gates, 25.4-KB on-chip memories, and 373-MB/s off-chip memory bandwidth while achieving a real-time video processing of 1080P@60Hz at the maximum clock frequency of 297-MHz. Moreover, the proposed VLSI circuits are fully pipelined and synchronized to the pixel clock of output video, so can be seamlessly integrated into diverse real-time video processing systems. [ABSTRACT FROM PUBLISHER]

The ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Moving Picture Experts Group (MPEG) issued in October 2017 a joint Call for Proposals (CfP) on video compression with capability beyond HEVC. The joint CfP included three categories of content: standard dynamic range (SDR), high dynamic range and wide color gamut (HDR/WCG), and 360° omni-directional video (360°). This paper describes a response to the joint CfP that considers all three categories of video content. The core codec in the response is designed based on the joint exploration model (JEM) reference software. The key coding tools in the JEM are significantly simplified to reduce both average and worst-case complexity for hardware design with negligible coding performance loss. Furthermore, two additional coding tools are used to further improve coding efficiency. For the HDR and 360° categories, additional coding tools specifically designed to optimize the compression efficiency and subjective quality of that specific content category are included. Further, some SDR coding tools are modified to alleviate subjective quality problems. For the random access configuration, compared to the HEVC test model (HM) anchor, the proposed video codec achieves average luma rate savings of 35.7%, 31.3%, and 33.9% for the SDR, HDR, and 360° categories, respectively. [ABSTRACT FROM AUTHOR]

Cloud computing provides reliable, affordable, and flexible resources for many applications and users with constrained computing resources and capabilities. The cloud computing concept is becoming an appealing paradigm for many industries including the gaming industry, leading to the introduction of cloud gaming architectures. Despite its advantages, cloud gaming suffers from unguaranteed end-to-end delay as well as server side’s computational complexity. In this paper, a novel algorithm for reducing the computational complexity and hence speeding up the video encoding speed is proposed. Specifically, by performing minimum modifications in the game engine and the video codec, some information from the game engine is fed into the video encoder to bypass the motion estimation (ME) process. Our results show that the proposed method achieves up to 39% speedup in the ME process, leading to a 24% acceleration in the total encoding process. [ABSTRACT FROM AUTHOR]

The promising potential and emerging applications of cloud gaming have drawn increasing interest from academia, industry, and the general public. However, providing a high-quality gaming experience in the cloud gaming framework is a challenging task because of the tradeoff between resource consumption and player emotion, which is affected by the game screen. We tackle this problem by leveraging emotion-aware screen effects in the cloud gaming framework and combining them with remote display technology. The first stage in the framework is the learning or training stage, which establishes a relationship between screen features and emotions using Gaussian mixture model-based classifiers. In the operating stage, a linear programming model provides appropriate screen changes based on the real-time user emotion obtained in the first stage. Our experiments demonstrate the effectiveness of the proposed framework. The results show that our proposed framework can provide a high quality gaming experience while generating an acceptable amount of workload for the cloud server in terms of resource consumption. [ABSTRACT FROM AUTHOR]

An image dehazing technique can restore hazy images to clear ones. Under hazy conditions, it can be a preceding stage of advanced-driver assistance systems (ADAS) to keep driving images clear. ADAS pursues a larger image resolution with a higher frame rate to keep its reliability for driving safety. This trend drives the image dehazing to face a severe challenge of considerable throughput. In this study, a hardware-efficient image dehazing engine with low computational burdens is proposed. It comprises two core techniques: Depth-Refinement Transmission-Rate Estimation (DTE) and Distribute Airlight Estimation (DAE). DTE provides an accurate depth map to estimate transmission rate, where the depth of white-objects is carefully calibrated. Base on the depth information, DAE predicts the airlight with the average of distant illumination sources to compensate nonuniform haze concentration. DTE and DAE can reduce the number of line buffers by 33% at least, while keeping comparable image dehazing quality. This study is implemented using TSMC CMOS $0.18~\mu \text{m}$ technology. To construct a smooth data scheduling, it is organized into a six-stage pipelinging strategy with an interleaving access technique. The logic gate count is 14.4K, and the power consumption is 15.2mW@250MHz. With a two-level parallelism, its throughput can support 3, $840\times 2.160$ @60fps. The experiment results show this study presents the superior combined performance of energy efficiency, area efficiency and line buffer reduction. [ABSTRACT FROM AUTHOR]

Long-term Video Question Answering plays an essential role in visual information retrieval, which aims at generating natural language answers to discretionary free-form questions about the referenced long-term video. Rather than remember the video as a sequence of visual content, humans have an innate cognitive ability to identify the critical moments related to the question at first glance, then tie together the specific evidence around these critical moments for further analysis and reasoning. Motivated by this intuition, we propose the multimodal hierarchical memory attentive networks with two heterogeneous memory subnetworks: the top guided memory network and the bottom enhanced multimodal memory attentive network. The top guided memory network serves as a shallow inference engine to pick relevant and informative moments of questions and obtain salient video content at a coarse-grained level. Subsequently, the bottom enhanced multimodal memory attentive network is designed as an in-depth reasoning engine to perform more accurate attention with cues from video bottom evidence in a fine-grained level to enhance question answering quality. We evaluate the proposed method on three publicly available video question answering benchmarks, namely ActivityNet-QA, MSRVTT-QA, and MSVD-QA. Experimental results demonstrate that the proposed approach significantly outperforms other state-of-the-art methods for long-term videos. Extensive ablation studies are carried out to explore the reasons behind the proposed model’s effectiveness. [ABSTRACT FROM AUTHOR]