Search

The paper develops datasets and methods to assess student participation in real-life collaborative learning environments. In collaborative learning environments, students are organized into small groups where they are free to interact within their group. Thus, students can move around freely causing issues with strong pose variation, move out and re-enter the camera scene, or face away from the camera. We formulate the problem of assessing student participation into two subproblems: (i) student group detection against strong background interference from other groups, and (ii) dynamic participant tracking within the group. A massive independent testing dataset of 12,518,250 student label instances, of total duration of 21 hours and 22 minutes of real-life videos, is used for evaluating the performance of our proposed method for student group detection. The proposed method of using multiple image representations is shown to perform equally or better than YOLO on all video instances. Over the entire dataset, the proposed method achieved an F1 score of 0.85 compared to 0.80 for YOLO. Following student group detection, the paper presents the development of a dynamic participant tracking system for assessing student group participation through long video sessions. The proposed dynamic participant tracking system is shown to perform exceptionally well, missing a student in just one out of 35 testing videos. In comparison, a state-of-the-art method fails to track students in 14 out of the 35 testing videos. The proposed method achieves 82.3% accuracy on an independent set of long, real-life collaborative videos.

The well-known 0–1 principle for traditional data-oblivious sorting networks states that a network with L inputs can be fully validated with an input vector set consisting of all $2^{L}$ unique vectors containing only the values 0 and 1 in each vector’s L inputs. Researchers providing proofs of this principle tend to ignore the fact, which is emphasized here, that 0–1 vectors provide all distinct orderings of the two inputs to the 2-sorters which perform all of the sorting operations in the networks. The authors have recently described single-stage N-sorters, with $N{>}2$ , and multiway merge sorting networks which use these N-sorters. The new N-sorters and their networks are also data-oblivious. It is easily shown that 0–1 vectors are not sufficient to fully verify even a single-stage 3-sorter, the smallest such N-sorter, as the 0–1 vectors are unable to produce all distinct orderings of the 3 inputs. In order to verify these N-sorters, the authors propose the 1-N principle, which states that testing an N-sorter with all distinct orderings of N input values is necessary and sufficient to prove the N-sorter’s correctness. An algorithm is defined which generates a vector set consisting of all distinct orderings of N inputs, which is then used to fully verify the associated single-stage N-sorter. In order to validate the authors’ L-input sorting networks which use these N-sorters, methods have been created which produce validation vector sets that are dramatically reduced in size versus the unsorted vector sets they are derived from. For example, the ratios of the number of L! permutation vectors to the equivalent reduced vectors are ${>}$ 1,000,000 for $\text{L}{=}12$ , and are much higher as L increases.

Monitoring disease evolution in Multiple sclerosis (MS) subjects may aid in decision making for personalizing treatment and disease evolution prediction. We investigate the use of disability progression, using clinical features, the expanded disability status scale (EDSS), and their relationship with texture features and Amplitude Modulation-Frequency Modulation (AM-FM) features extracted from MRI MS detectable lesions for the prognosis of future disability on magnetic resonance imaging (MRI). MS detectable brain lesions from N=38 symptomatic untreated subjects diagnosed with clinically isolated syndrome (CIS), were manually segmented, by an experienced MS neurologist, on transverse T2-weighted (T2W) images obtained from serial brain MRI scans at the baseline (Time0M) and the repeat (Time16-12M) examinations. The subjects were separated into two different groups based on their EDSS: (G1: $1\le $ EDSS $_{\mathrm {2Y}}\le 3.5$ (N=26) and G2: $3.5 < $ EDSS $_{\mathrm {2Y}}\le 8.5$ (N=12) and were monitored over ten years’ time (Time $_{\mathrm {10Y}}$ ). After intensity normalization and image registration, texture and AM-FM features were extracted from all MS lesions at Time0M and Time16-12M. The extracted features were used to develop models that correlated with the disease progression in Time10Y. We found statistically significant differences for features extracted from the two different groups (G1 vs G2 at Time $_{\mathrm {10Y}}$ ) and these might be used to predict the development and or the severity of the MS disease. The best model for classifying G1 vs G2 subjects at Time10Y included information taken from the MS lesion images, texture features and AM-FM features extracted from those MS lesion images (with a correct classification score of %CC=94). The proposed methodology may contribute to additional factors for predicting the development and assessing the severity of the MS disease. However, a larger scale study is needed to establish the application in clinical practice and for computing additional features that may provide information for better and earlier differentiation between normal tissue and MS lesions.

Speaker diarization refers to methods for identifying speakers from audio recordings. An important application comes from the need to assess student interactions in collaborative learning environments. Diarization is very difficult in these environments where a single microphone is used to record multiple voices. Although there have been advancements in this field, little progress has been made in the case of noisy and disorganized multi-speaker environments. Current state-of-the-art methods based on Deep Learning require large training databases and can suffer from significant noise interference and bias due to the speaker’s accent, age, and gender. In this paper, we are proposing a new method to identify speakers that does not require the use of large training sets. To this end, we use a virtual array of microphones. The signal at the virtual microphones is simulated by extracting the spatial information of the speakers from a single channel audio recording using approximate speaker geometry observed from a video recording. The Room Impulse Responses (RIRs) at the virtual microphones are then estimated using acoustic scene simulations. The RIRs are then used to compute a cross-correlation matrix of possible audio sources. Speaker diarization is performed using the cross-correlation matrices as input to a classifier. For the task of identifying active student speakers in classroom audio, the proposed method significantly outperformed diarization methods performed by Google Cloud and Amazon AWS services.

In hardware such as FPGAs, Kenneth Batcher’s Odd-Even Merge Sort and Bitonic Merge Sort are the state-of-the-art methodologies used to quickly sort a list of more than 16 input values. Both sorting networks feature merges of 2 sorted input lists into a single sorted output list. For both, a full sort of 64 and 512 input values requires 21 and 45 serial stages, respectively. Multiway merge sorting networks described here require significantly fewer serial stages. For example, 8-way merge networks fully sort 64 and 512 input values in 9 and 20 serial stages, less than half the number of the respective 2-way networks. When the multiway merge sorting networks utilize the single-stage $N$ -sorters recently defined by the authors, they are considerably faster than Batcher’s networks. In the AMD-Xilinx Ultrascale+ xcvu9p FPGA, the two 8-way merge networks have speedups of 1.85 and 1.74 versus the comparable 2-way networks. A fully pipelined 3-way merge network in this FPGA is capable of fully sorting 500 million lists of 729 unsorted 32-bit values in one second. In software, multiway merge methods are used to find the median of certain pixel rectangles in images, since the median can be determined in fewer stages than are required to fully sort the rectangle. However, the software still requires a series of many 2-sorter operations to find a median. These multiway merge median methods are dramatically sped up in hardware, where the authors’ new single-stage $N$ -sorters and $N$ -filters operate in parallel in each stage of the merge process.

The authors’ recently published design system for the creation of single-stage N-sorter/N-filter sorting devices, which were implemented using a particular example hardware block, is here expanded and applied to a second hardware type, FPGA carry chain logic. Although several researchers have published applications which use FPGA carry chain logic, most do not use carry chain logic as is done here, and none of the applications target sorting devices. Prior to the introduction of the single-stage N-sorter/N-filter design system, the fastest state-of-the-art hardware devices which sorted more than 2 input values were multistage sorting networks, in which the sorting process is performed by one or more 2-sorters and 2-max/2-min filters, operating in each sequential stage. Using the authors’ original design system, single-stage N-sorters and N-filters were shown to be faster than the fastest comparable sorting networks when sorting 3 to 9 inputs. Here, product term splitting and a new Sum-of-Products output multiplexer equation are added to the design system, and this expanded design system is then implemented in carry chain logic to build faster and larger N-sorters, and much larger and still fast N-max/N-min filters. The new carry chain N-sorters are implemented in the FPGA used in the Amazon AWS EC2 F1 instance, which is one of the two example FPGAs utilized in the previous publication. A carry chain logic 16-sorter, not practical when using the original hardware block, has a speedup of 4.61 relative to the fastest 16-network. An example of the new, very large single-stage carry chain N-max filters is the 125-max $5\times 5\times 5$ CNN video max pooling filter, which operates in only 2.075 nS. A 2-stage 1024-max network, using single-stage 32-max carry chain filters, has a speedup of 2.85 versus the existing state-of-the-art 10-stage network of 2-max filters.

There is strong interest in developing high-performance hardware sorting systems which can sort a set of elements as quickly as possible. The fastest of the current FPGA systems are sorting networks, in which sets of 2-sorters operate in parallel in each series stage of a multi-stage sorting process. A 2-sorter is a single-stage hardware block which sorts two values, so any list with more than 2 values must be sorted with a series network of 2-sorters. A primary contribution of this work is to provide a general methodology for the design of stable single-stage hardware sorters which sort more than 2 values simultaneously. This general methodology for $N$ -sorter design, with $N{>}2$ , is then adapted for use in modern FPGAs, where it is shown that single-stage 3-sorters up to 9-sorters have speedup ratios from 2.0 to 3.5 versus the comparable state-of-the-art 2-sorter networks. A design system modification is shown to produce even faster single-stage $N$ -max and $N$ -min filters. When used for max pooling 32-bit data in the fastest analyzed FPGA, a single 9-max filter will process 500 million 9-pixel groups per second (4K:3840x2160 at 500 frames/second). The single-stage 9-median filter using this design methodology, useful in image processing, is shown to have speedup ratios of 3.0 to 4.1 versus state-of-the-art FPGA network implementations, even though its resource usage is comparable to, often better than, the network implementations. Ten 8-bit 9-median filters operating in parallel in the fastest FPGA will process over 5.4 billion pixels/sec (4K at over 600 frames/second).

Video compression is the core technology in mobile (mHealth) and electronic (eHealth) health video streaming applications. With global video traffic projected to reach 82% of all Internet traffic by 2022, there is a strong need to develop efficient compression algorithms to accommodate expected future growth. For the first time in decades, and especially since ISO/IEC MPEG and ITU-T VCEG expert groups strategically joined forces to develop the highly successful H.264/AVC standard, we have two distinct initiatives competing for the best performing video codec. On the one hand, we have the Alliance for Open Media (AOM) that support a new, royalty free video codec generation, termed AV1, based on VP8 and VP9 efforts. On the other hand, the Joint Video Exploration Team (JVET) has been developing the Versatile Video Codec (VVC) as the successor of the High Efficiency Video Coding (HEVC) standard. At the same time, the breadth of applications utilizing video codecs, involving significant content variability and moving across the video resolution ladder, to satisfy different constraints, have resulted in mixed literature results, with respect to the best performing codec. In this paper, we compare the performance of emerging VVC and AV1 codecs, along with popular HEVC implementations, namely the HEVC Test Model (HM) and x265, as well as earlier, VP9 codec, and investigate their suitability for medical applications. To the best of our knowledge, this is the first performance comparison of emerging VVC and AV1 video codecs for use in the healthcare domain. Experimental evaluation based on three datasets (ultrasound, emergency scenery, and general-purpose videos) demonstrate that VVC outperforms all rival codecs while AV1 achieves better compression efficiency than HEVC in all cases but low-resolution ($560\times448$ @40Hz) ultrasound videos of the common carotid artery. Furthermore, the use of video despeckling prior to ultrasound video compression can provide significant bitrate savings.

Dynamically reconfigurable computing platforms provide promising methods for dynamic management of hardware resources, power, and performance. Yet, progress in dynamically reconfigurable computing is fundamentally limited by the reconfiguration time overhead. Prior research in the development of dynamic partial reconfiguration (DPR) controllers has been limited by its use of the Processor Local Bus (PLB). As a result, the bus was unavailable during DPR. This resulted in significant time overhead. To minimize the overhead, we introduce the use of a multiport memory controller (MPMC) that frees the PLB during the reconfiguration process. The processor is thus allowed to switch to other tasks during the reconfiguration operation. This effectively limits the reconfiguration overhead. An interrupt is used to inform the processor when the operation is complete. Therefore, the system can multitask during the reconfiguration operation. Furthermore, to maximize performance, we introduce the use of overclocking with active feedback. During overclocking, the use of active feedback is used to ensure that the device voltage and temperature are within nominal operating conditions. All of these contributions lead to significant performance improvements over current partial reconfiguration subsystems. The portability of the system, demonstrated on the Virtex-4 and the Virtex-5, consists of four different hardware platforms.