Search

Semi-supervised semantic segmentation aims to learn from a small amount of labeled data and plenty of unlabeled ones for the segmentation task. The most common approach is to generate pseudo-labels for unlabeled images to augment the training data. However, the noisy pseudo-labels will lead to cumulative classification errors and aggravate the local inconsistency in prediction. This paper proposes a Region Relevance Network (RRN) to alleviate the problem mentioned above. Specifically, we first introduce a local pseudo-label filtering module that leverages discriminator networks to assess the accuracy of the pseudo-label at the region level. A local selection loss is proposed to mitigate the negative impact of wrong pseudo-labels in consistency regularization training. In addition, we propose a dynamic region-loss correction module, which takes the merit of network diversity to further rate the reliability of pseudo-labels and correct the convergence direction of the segmentation network with a dynamic region loss. Extensive experiments are conducted on PASCAL VOC 2012 and Cityscapes datasets with varying amounts of labeled data, demonstrating that our proposed approach achieves state-of-the-art performance compared to current counterparts., Comment: accepted by IEEE International Conference on Multimedia and Expo 2023

This paper studies real-time collaborative robot (cobot) handling, where the cobot maneuvers an object under human dynamic gesture commands. Enabling dynamic gesture commands is useful when the human needs to avoid direct contact with the robot or the object handled by the robot. However, the key challenge lies in the heterogeneity in human behaviors and the stochasticity in the perception of dynamic gestures, which requires the robot handling policy to be adaptable and robust. To address these challenges, we introduce Conditional Collaborative Handling Process (CCHP) to encode a contextaware cobot handling policy and a procedure to learn such policy from human-human collaboration. We thoroughly evaluate the adaptability and robustness of CCHP and apply our approach to a real-time cobot assembly task with Kinova Gen3 robot arm. Results show that our method leads to significantly less human effort and smoother human-robot collaboration than state-of-the-art rule-based approach even with first-time users., Comment: IEEE Robotics and Automation Letters (RA-L), 2023. arXiv admin note: substantial text overlap with arXiv:2112.06020

In topological insulators, massive surface bands resulting from local symmetry breaking are believed to exhibit a half-quantized Hall conductance. However, such scenarios are obviously inconsistent with the Thouless-Kohmoto-Nightingale-Nijs theorem, which states that a single band in a lattice with a finite Brillouin zone can only have an integer-quantized Hall conductance. To explore this, we investigate the band structures of a lattice model describing the magnetic topological insulator film that supports the axion insulator, Chern insulator, and semi-magnetic topological insulator phases. We reveal that the gapped and gapless surface bands in the three phases are characterized by an integer-quantized Hall conductance and a half-quantized Hall conductance, respectively. This result is distinct from the previous consensus that the gapped surface band is responsible for the half-quantized Hall conductance and the gapless band should exhibit zero Hall response. We propose an effective model to describe the three phases and show that the low-energy dispersion of the surface bands inherits from the surface Dirac fermions. The gapped surface band manifests a nearly half-quantized Hall conductance at low energy near the center of Brillouin zone, but is compensated by another nearly half-quantized Hall conductance at high energy near the boundary of Brillouin zone because a single band can only have an integer-quantized Hall conductance. The gapless state hosts a zero Hall conductance at low energy but is compensated by another half-quantized Hall conductance at high energy, and thus the half-quantized Hall conductance can only originate from the gapless band. Moreover, we calculate the layer-resolved Hall conductance of the system. The conclusion suggests that the individual gapped surface band alone does not support the half-quantized surface Hall effect in a lattice model.

Automatic 3D content creation has achieved rapid progress recently due to the availability of pre-trained, large language models and image diffusion models, forming the emerging topic of text-to-3D content creation. Existing text-to-3D methods commonly use implicit scene representations, which couple the geometry and appearance via volume rendering and are suboptimal in terms of recovering finer geometries and achieving photorealistic rendering; consequently, they are less effective for generating high-quality 3D assets. In this work, we propose a new method of Fantasia3D for high-quality text-to-3D content creation. Key to Fantasia3D is the disentangled modeling and learning of geometry and appearance. For geometry learning, we rely on a hybrid scene representation, and propose to encode surface normal extracted from the representation as the input of the image diffusion model. For appearance modeling, we introduce the spatially varying bidirectional reflectance distribution function (BRDF) into the text-to-3D task, and learn the surface material for photorealistic rendering of the generated surface. Our disentangled framework is more compatible with popular graphics engines, supporting relighting, editing, and physical simulation of the generated 3D assets. We conduct thorough experiments that show the advantages of our method over existing ones under different text-to-3D task settings. Project page and source codes: https://fantasia3d.github.io/., Comment: Accepted by ICCV 2023. Project page: https://fantasia3d.github.io/

In this paper we prove that Ext-analogues of Gan--Gross--Prasad models vanish for tempered representations, as conjectured by D. Prasad. In particular, this implies a conjectural Euler--Poincar\'e characteristic formula for the Gan--Gross--Prasad models., Comment: 17 pages, comments welcome!

Fairness in machine learning has attracted increasing attention in recent years. The fairness methods improving algorithmic fairness for in-distribution data may not perform well under distribution shifts. In this paper, we first theoretically demonstrate the inherent connection between distribution shift, data perturbation, and model weight perturbation. Subsequently, we analyze the sufficient conditions to guarantee fairness (i.e., low demographic parity) for the target dataset, including fairness for the source dataset, and low prediction difference between the source and target datasets for each sensitive attribute group. Motivated by these sufficient conditions, we propose robust fairness regularization (RFR) by considering the worst case within the model weight perturbation ball for each sensitive attribute group. We evaluate the effectiveness of our proposed RFR algorithm on synthetic and real distribution shifts across various datasets. Experimental results demonstrate that RFR achieves better fairness-accuracy trade-off performance compared with several baselines. The source code is available at \url{https://github.com/zhimengj0326/RFR_NeurIPS23}., Comment: NeurIPS 2023

Accurately manipulating articulated objects is a challenging yet important task for real robot applications. In this paper, we present a novel framework called Sim2Real$^2$ to enable the robot to manipulate an unseen articulated object to the desired state precisely in the real world with no human demonstrations. We leverage recent advances in physics simulation and learning-based perception to build the interactive explicit physics model of the object and use it to plan a long-horizon manipulation trajectory to accomplish the task. However, the interactive model cannot be correctly estimated from a static observation. Therefore, we learn to predict the object affordance from a single-frame point cloud, control the robot to actively interact with the object with a one-step action, and capture another point cloud. Further, the physics model is constructed from the two point clouds. Experimental results show that our framework achieves about 70% manipulations with <30% relative error for common articulated objects, and 30% manipulations for difficult objects. Our proposed framework also enables advanced manipulation strategies, such as manipulating with different tools. Code and videos are available on our project webpage: https://ttimelord.github.io/Sim2Real2-site/, Comment: Accepted to IEEE International Conference on Robotics and Automation (ICRA) 2023

We perform a study on the generalization ability of the wide two-layer ReLU neural network on $\mathbb{R}$. We first establish some spectral properties of the neural tangent kernel (NTK): $a)$ $K_{d}$, the NTK defined on $\mathbb{R}^{d}$, is positive definite; $b)$ $\lambda_{i}(K_{1})$, the $i$-th largest eigenvalue of $K_{1}$, is proportional to $i^{-2}$. We then show that: $i)$ when the width $m\rightarrow\infty$, the neural network kernel (NNK) uniformly converges to the NTK; $ii)$ the minimax rate of regression over the RKHS associated to $K_{1}$ is $n^{-2/3}$; $iii)$ if one adopts the early stopping strategy in training a wide neural network, the resulting neural network achieves the minimax rate; $iv)$ if one trains the neural network till it overfits the data, the resulting neural network can not generalize well. Finally, we provide an explanation to reconcile our theory and the widely observed ``benign overfitting phenomenon''., Comment: 47 pages, 4 figures

In practical recommendation scenarios, users often interact with items under multi-typed behaviors (e.g., click, add-to-cart, and purchase). Traditional collaborative filtering techniques typically assume that users only have a single type of behavior with items, making it insufficient to utilize complex collaborative signals to learn informative representations and infer actual user preferences. Consequently, some pioneer studies explore modeling multi-behavior heterogeneity to learn better representations and boost the performance of recommendations for a target behavior. However, a large number of auxiliary behaviors (i.e., click and add-to-cart) could introduce irrelevant information to recommenders, which could mislead the target behavior (i.e., purchase) recommendation, rendering two critical challenges: (i) denoising auxiliary behaviors and (ii) bridging the semantic gap between auxiliary and target behaviors. Motivated by the above observation, we propose a novel framework-Denoising and Prompt-Tuning (DPT) with a three-stage learning paradigm to solve the aforementioned challenges. In particular, DPT is equipped with a pattern-enhanced graph encoder in the first stage to learn complex patterns as prior knowledge in a data-driven manner to guide learning informative representation and pinpointing reliable noise for subsequent stages. Accordingly, we adopt different lightweight tuning approaches with effectiveness and efficiency in the following stages to further attenuate the influence of noise and alleviate the semantic gap among multi-typed behaviors. Extensive experiments on two real-world datasets demonstrate the superiority of DPT over a wide range of state-of-the-art methods. The implementation code is available online at https://github.com/zc-97/DPT., Comment: Proceedings of the ACM Web Conference 2023 (WWW '23)

Generalizable manipulation skills, which can be composed to tackle long-horizon and complex daily chores, are one of the cornerstones of Embodied AI. However, existing benchmarks, mostly composed of a suite of simulatable environments, are insufficient to push cutting-edge research works because they lack object-level topological and geometric variations, are not based on fully dynamic simulation, or are short of native support for multiple types of manipulation tasks. To this end, we present ManiSkill2, the next generation of the SAPIEN ManiSkill benchmark, to address critical pain points often encountered by researchers when using benchmarks for generalizable manipulation skills. ManiSkill2 includes 20 manipulation task families with 2000+ object models and 4M+ demonstration frames, which cover stationary/mobile-base, single/dual-arm, and rigid/soft-body manipulation tasks with 2D/3D-input data simulated by fully dynamic engines. It defines a unified interface and evaluation protocol to support a wide range of algorithms (e.g., classic sense-plan-act, RL, IL), visual observations (point cloud, RGBD), and controllers (e.g., action type and parameterization). Moreover, it empowers fast visual input learning algorithms so that a CNN-based policy can collect samples at about 2000 FPS with 1 GPU and 16 processes on a regular workstation. It implements a render server infrastructure to allow sharing rendering resources across all environments, thereby significantly reducing memory usage. We open-source all codes of our benchmark (simulator, environments, and baselines) and host an online challenge open to interdisciplinary researchers., Comment: Published as a conference paper at ICLR 2023. Project website: https://maniskill2.github.io/

Despite the tremendous success of Reinforcement Learning (RL) algorithms in simulation environments, applying RL to real-world applications still faces many challenges. A major concern is safety, in another word, constraint satisfaction. State-wise constraints are one of the most common constraints in real-world applications and one of the most challenging constraints in Safe RL. Enforcing state-wise constraints is necessary and essential to many challenging tasks such as autonomous driving, robot manipulation. This paper provides a comprehensive review of existing approaches that address state-wise constraints in RL. Under the framework of State-wise Constrained Markov Decision Process (SCMDP), we will discuss the connections, differences, and trade-offs of existing approaches in terms of (i) safety guarantee and scalability, (ii) safety and reward performance, and (iii) safety after convergence and during training. We also summarize limitations of current methods and discuss potential future directions., Comment: IJCAI Survey Track 2023

Multi-agent motion planning (MAMP) is a critical challenge in applications such as connected autonomous vehicles and multi-robot systems. In this paper, we propose a space-time conflict resolution approach for MAMP. We formulate the problem using a novel, flexible sphere-based discretization for trajectories. Our approach leverages a depth-first conflict search strategy to provide the scalability of decoupled approaches while maintaining the computational guarantees of coupled approaches. We compose procedures for evading discretization error and adhering to kinematic constraints in generated solutions. Theoretically, we prove the continuous-time feasibility and formulation-space completeness of our algorithm. Experimentally, we demonstrate that our algorithm matches the performance of the current state of the art with respect to both runtime and solution quality, while expanding upon the abilities of current work through accommodation for both static and dynamic obstacles. We evaluate our algorithm in various unsignalized traffic intersection scenarios using CARLA, an open-source vehicle simulator. Results show significant success rate improvement in spatially constrained settings, involving both connected and non-connected vehicles. Furthermore, we maintain a reasonable suboptimality ratio that scales well among increasingly complex scenarios.

Complex vector modes, entangled in spin and orbital angular momentum, are opening burgeoning opportunities for a wide variety of applications. Importantly, the flexible manipulation the various properties of such beams will pave the way to novel applications. As such, in this manuscript, we demonstrate a longitudinal spin-orbit separation of complex vector modes propagating in free space. To achieve this we employed the recently demonstrated circular Airy Gaussian vortex vector (CAGVV) modes, which feature a self-focusing property. More precisely, by properly manipulating the intrinsic parameters of CAGVV modes, the strong coupling between the two constituting orthogonal components of CAGVV mode undergo a spin-orbit separation along the propagation direction namely, while one polarisation component, focuses at a specific plane, the other focuses at a different plane. Such spin-orbit separation, which we demonstrated by numerical simulations and corroborated experimentally, can be adjusted on-demand by simply changing the initial parameters of CAGVV modes. Our findings will be of great relevance, for example in optical tweezers, to manipulate micro- or nano-particles at two different parallel planes.

Dantzig-Wolfe (DW) decomposition is a well-known technique in mixed-integer programming (MIP) for decomposing and convexifying constraints to obtain potentially strong dual bounds. We investigate cutting planes that can be derived using the DW decomposition algorithm and show that these cuts can provide the same dual bounds as DW decomposition. More precisely, we generate one cut for each DW block, and when combined with the constraints in the original formulation, these cuts imply the objective function cut one can simply write using the DW bound. This approach typically leads to a formulation with lower dual degeneracy that consequently has a better computational performance when solved by standard MIP solvers in the original space. We also discuss how to strengthen these cuts to improve the computational performance further. We test our approach on the Multiple Knapsack Assignment Problem and the Temporal Knapsack Problem, and show that the proposed cuts are helpful in accelerating the solution time without the need to implement branch and price.

Photonic integrated circuits (PICs) have the potential to drastically expand the capabilities of optical communications, sensing, and quantum information science and engineering. However, PICs are commonly fabricated using selective material etching, a subtractive process. Thus, the chip's functionality cannot be substantially altered once fabricated. Here, we propose to exploit wide-bandgap non-volatile phase-change materials (PCMs) to create a rewritable PIC platform. A PCM-based PIC can be written using a nano-second pulsed laser without removing any material, akin to rewritable compact disks. The whole circuit can then be erased by heating, and a completely new circuit can be rewritten. We designed a dielectric-assisted PCM waveguide consisting of a thick dielectric layer on top of a thin layer of wide-bandgap PCMs Sb2S3 and Sb2Se3. The low-loss PCMs and our engineered waveguiding structure lead to a negligible optical loss. Furthermore, we analyzed and specified the spatio-temporal laser pulse shape to write the PCMs. Our proposed platform will enable low-cost manufacturing and have a far-reaching impact on the rapid prototyping of PICs, validation of new designs, and photonic education.

Theoretically, the quantum anomalous Hall effect is characterized by a quantized Hall conductance. However, many experiments only reported the quantization of the Hall resistance, which is accompanied by a non-vanishing longitudinal resistance, resulting in a non-quantized Hall conductance. Meanwhile, the non-vanishing longitudinal resistance features a universal exponential decay with the increase in magnetic field. Such a discrepancy obviously challenges the understanding of the quantum anomalous Hall effect. To this end, we propose that the coupling of chiral edge states, which has not been properly evaluated in the previous theories, hinders the quantization of the Hall conductance, while it maintains the quantization of the Hall resistance. The coupling between the chiral edges states along the opposite boundaries can be assisted by magnetic domains or defects inside the sample bulk, which has been already identified in recent experiments. We demonstrate that the chiral edge state coupling theory works rather well to explain the experimental results in both magnetic topological insulator and moir\'e superlattice systems.

Scalable programmable photonic integrated circuits (PICs) can potentially transform the current state of classical and quantum optical information processing. However, traditional means of programming, including thermo-optic, free carrier dispersion, and Pockels effect result in either large device footprints or high static energy consumptions, significantly limiting their scalability. While chalcogenide-based non-volatile phase-change materials (PCMs) could mitigate these problems thanks to their strong index modulation and zero static power consumption, they often suffer from large absorptive loss, low cyclability, and lack of multilevel operation. Here, we report a wide-bandgap PCM antimony sulfide (Sb2S3)-clad silicon photonic platform simultaneously achieving low loss, high cyclability, and 5-bit operation. We switch Sb2S3 via an on-chip silicon PIN diode heater and demonstrate components with low insertion loss (<1.0 dB), high extinction ratio (>10 dB), and high endurance (>1,600 switching events). Remarkably, we find that Sb2S3 can be programmed into fine intermediate states by applying identical and thermally isolated pulses, providing a unique approach to controllable multilevel operation. Through dynamic pulse control, we achieve on-demand and accurate 5-bit (32 levels) operations, rendering 0.50 +- 0.16 dB contrast per step. Using this multilevel behavior, we further trim random phase error in a balanced Mach-Zehnder interferometer. Our work opens an attractive pathway toward non-volatile large-scale programmable PICs with low-loss and on-demand multi-bit operations., Comment: 22 pages, 6 figures in main text

Graph neural networks (GNNs) have received remarkable success in link prediction (GNNLP) tasks. Existing efforts first predefine the subgraph for the whole dataset and then apply GNNs to encode edge representations by leveraging the neighborhood structure induced by the fixed subgraph. The prominence of GNNLP methods significantly relies on the adhoc subgraph. Since node connectivity in real-world graphs is complex, one shared subgraph is limited for all edges. Thus, the choices of subgraphs should be personalized to different edges. However, performing personalized subgraph selection is nontrivial since the potential selection space grows exponentially to the scale of edges. Besides, the inference edges are not available during training in link prediction scenarios, so the selection process needs to be inductive. To bridge the gap, we introduce a Personalized Subgraph Selector (PS2) as a plug-and-play framework to automatically, personally, and inductively identify optimal subgraphs for different edges when performing GNNLP. PS2 is instantiated as a bi-level optimization problem that can be efficiently solved differently. Coupling GNNLP models with PS2, we suggest a brand-new angle towards GNNLP training: by first identifying the optimal subgraphs for edges; and then focusing on training the inference model by using the sampled subgraphs. Comprehensive experiments endorse the effectiveness of our proposed method across various GNNLP backbones (GCN, GraphSage, NGCF, LightGCN, and SEAL) and diverse benchmarks (Planetoid, OGB, and Recommendation datasets). Our code is publicly available at \url{https://github.com/qiaoyu-tan/PS2}

Real-time bidding (RTB) has become a major paradigm of display advertising. Each ad impression generated from a user visit is auctioned in real time, where demand-side platform (DSP) automatically provides bid price usually relying on the ad impression value estimation and the optimal bid price determination. However, the current bid strategy overlooks large randomness of the user behaviors (e.g., click) and the cost uncertainty caused by the auction competition. In this work, we explicitly factor in the uncertainty of estimated ad impression values and model the risk preference of a DSP under a specific state and market environment via a sequential decision process. Specifically, we propose a novel adaptive risk-aware bidding algorithm with budget constraint via reinforcement learning, which is the first to simultaneously consider estimation uncertainty and the dynamic risk tendency of a DSP. We theoretically unveil the intrinsic relation between the uncertainty and the risk tendency based on value at risk (VaR). Consequently, we propose two instantiations to model risk tendency, including an expert knowledge-based formulation embracing three essential properties and an adaptive learning method based on self-supervised reinforcement learning. We conduct extensive experiments on public datasets and show that the proposed framework outperforms state-of-the-art methods in practical settings., Comment: Accepted by AI for Web Advertising Workshop, AAAI 2023

In this work, we theoretically investigate the light-induced topological phases and finite-size crossovers in a paradigmatic quantum spin Hall (QSH) system with high-frequency pumping optics. Taking the HgTe quantum well for an example, our numerical results show that circularly polarized light can break time-reversal symmetry and induce the quantum anomalous Hall (QAH) phase. In particular, the coupling between the edge states is spin dependent and is related not only to the size of the system, but also to the strength of the polarized pumping optics. By tuning the two parameters (system width and optical pumping strength), we obtain four transport regimes, namely, QSH, QAH, edge conducting, and normal insulator. These four different transport regimes have contrasting edge conducting properties, which will feature prominently in transport experiments on various topological materials., Comment: 16 pages, 6 figures, update references, published version

We study the online decision making problem (ODMP) as a natural generalization of online linear programming. In ODMP, a single decision maker undertakes a sequence of decisions over $T$ time steps. At each time step, the decision maker makes a locally feasible decision based on information available up to that point. The objective is to maximize the accumulated reward while satisfying some convex global constraints called goal constraints. The decision made at each step results in an $m$-dimensional vector that represents the contribution of this local decision to the goal constraints. In the online setting, these goal constraints are soft constraints that can be violated moderately. To handle potential nonconvexity and nonlinearity in ODMP, we propose a Fenchel dual-based online algorithm. At each time step, the algorithm requires solving a potentially nonconvex optimization problem over the local feasible set and a convex optimization problem over the goal set. Under certain stochastic input models, we show that the algorithm achieves $O(\sqrt{mT})$ goal constraint violation deterministically, and $\tilde{O}(\sqrt{mT})$ regret in expected reward. Numerical experiments on an online knapsack problem and an assortment optimization problem are conducted to demonstrate the potential of our proposed online algorithm.

The complexity class DP is the class of all languages that are the intersection of a language in NP and a language in co-NP, as coined by Papadimitriou and Yannakakis (1982). Hartvigsen and Zemel (1992) conjectured that recognizing a facet for the knapsack polytope is DP-complete. While it has been known that the recognition problems of facets for polytopes associated with other well-known combinatorial optimization problems, e.g., traveling salesman, node/set packing/covering, are DP-complete, this conjecture on recognizing facets for the knapsack polytope remains open. We provide a positive answer to this conjecture. Moreover, despite the DP-hardness of the recognition problem, we give a polynomial time algorithm for deciding if an inequality with a fixed number of distinct positive coefficients defines a facet of a knapsack polytope, generalizing a result of Balas (1975).