Search

The swift proliferation of edge intelligence and ubiquitous data generation have heightened privacy into a pressing societal need. State-of-the-art reversible privacy protection requires significant hardware resources at the edge with distinct architecture for sensors and security, leading to a rise in hardware overhead and expanded attack surfaces. To address these challenges, we propose a time-domain mixed-signal (TD-MS) circuit architecture facilitating in-situ privacy (ISP) with hardware-secured data reversibility. The proposed TD-MS ISP unites data acquisition, data conversion, key generation, and protection while providing authorized device-specific unclonable data recovery for forensic purposes. At the system level, we demonstrate the attack resilience and privacy-preserving computation performance by implementing a custom embedded system applied to real-world surveillance scenarios. At the circuit level, we showcase custom TD-MS circuits, evaluating their energy and area efficiency against a digital baseline implemented in 65nm technology. With full-stack SPICE simulations for both the baseline digital and proposed TD-MS circuits, we measured a $670\times $ energy/frame savings against the embedded system, $3\times $ area reduction and $3.2\times $ energy TD-MS gains over digital.

Near-eye gaze estimation is a task that maps the recording of an eye captured by an adjacent camera to the direction of a person's gaze in space. In contrast to frame-based cameras, event cameras are characterized by high sensing rates, low latency, sparse asynchronous data outputs, and high dynamic range, which are well suited for recording the fast eye movements. However, algorithms and system designs that operate on frame-based cameras are not applicable to event-based data, due to the natural differences in the data characteristics. In this work, we study the pattern of near-eye event-based data streams and extract eye features to estimate gaze. First, by analyzing eye parts and movements, and harnessing the polar, spatial, and temporal distribution of the events, we introduce a real-time pipeline to extract pupil features. Second, we present a recurrent neural network with a proposed coordinate-to-angle loss function to accurately estimate gaze from pupil feature sequence. We demonstrated that our system achieves accurate real-time estimation with angular accuracy of 0.46$^\circ$∘ and update rates of 950 Hz, thus opening up avenues for novel applications. To our knowledge, this is the first system that operates only on event-based data to perform gaze estimation.

Accurate identification of the target and tracking it at high speeds using drone-mounted cameras and compute hardware finds military and commercial applications. Conventional frame-based cameras and convolutional neural networks (CNNs) extract detailed spatial information to show high accuracy but suffer from lower throughput caused by large models. Alternatively, event cameras capture the motion information as an asynchronous event stream with high temporal resolution. Spiking neural networks (SNNs) can be used to process these data at high speed, but the sparse sensing and difficulty in training SNN limit the accuracy. Fusing the complementary spatial and temporal advantages of the frame and event-based pipelines allows high-speed identification and tracking while preserving accuracy. The SNN processes the event stream continuously to provide high-speed target estimates with lower accuracy, while periodic anchors provided by the reliable CNN restore the accuracy. In this work, we present a heterogeneous programmable ARM Cortex-based system-on-a-chip (SoC) in 40-nm Taiwan Semiconductor Manufacturing Company (TSMC) ultra low power (ULP) technology with power-efficient RRAM compute-in-memory (CIM) for CNN and high-speed SRAM compute-near-memory (CNM) for SNN for the modality-matched acceleration of the hybrid vision. Our SoC incorporates: 1) two levels of power gating to save 91.8% of total chip power with non-volatile RRAM-CIM; 2) embedded triple error correction (TEC) within RRAM CIM macro to suppress the raw bit errors in reading by >5 orders of magnitude; and 3) parallelly operating CNN and SNN modules to provide >100 outputs/s. Such cross-layer hybrid approaches can mitigate fundamental tradeoffs in sensing and processing.