Search

Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance. Furthermore, extracting pretrained embeddings for all modalities is computationally inefficient for inference. In this work, to achieve high efficiency-performance multimodal transfer learning, we propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model (teacher) to a specific modal fundamental model (student). With an intuition that the best learning performance comes with professional advisers and smart students, we use a CLIP-based teacher model to provide expressive multimodal knowledge supervision signals to a RoBERTa-based student model via optimizing a step-distillation objective loss—first step: the teacher distills multimodal knowledge of video-enhanced prompts from classification logits to a regression logit—second step: the multimodal knowledge is distilled from the regression logit of the teacher to the student. We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis (MOSI and MOSEI datasets) and audio-visual retrieval (VEGAS dataset). The student (requiring only the text modality as input) achieves an MAE score improvement of up to 12.3% for MOSI and MOSEI. Our method further enhances the state-of-the-art method by 3.4% mAP score for VEGAS without additional computations for inference. These results suggest the strengths of our method for achieving high efficiency-performance multimodal transfer learning.

Video-level sentiment analysis is a challenging task and requires systems to obtain discriminative multimodal representations that can capture difference in sentiments across various modalities. However, due to diverse distributions of various modalities and the unified multimodal labels are not always adaptable to unimodal learning, the distance difference between unimodal representations increases, and prevents systems from learning discriminative multimodal representations. In this paper, to obtain more discriminative multimodal representations that can further improve systems’ performance, we propose a VAE-based adversarial multimodal domain transfer (VAE-AMDT) and jointly train it with a multi-attention module to reduce the distance difference between unimodal representations. We first perform variational autoencoder (VAE) to make visual, linguistic and acoustic representations follow a common distribution, and then introduce adversarial training to transfer all unimodal representations to a joint embedding space. As a result, we fuse various modalities on this joint embedding space via the multi-attention module, which consists of self-attention, cross-attention and triple-attention for highlighting important sentimental representations over time and modality. Our method improves F1-score of the state-of-the-art by 3.6% on MOSI and 2.9% on MOSEI datasets, and prove its efficacy in obtaining discriminative multimodal representations for video-level sentiment analysis.

Abstract Background Exposure to radiation during catheter ablation procedures poses a risk to the heath of both the patient and electrophysiology laboratory staff. Recently, the feasibility and effectiveness of zero‐fluoroscopy ablation have been reported. However, studies on the outcomes of zero‐fluoroscopy ablation in Japan remain limited. This study investigated the outcomes of zero‐fluoroscopy ablation for cardiac arrhythmias at a Japanese institute. Methods and Results We present a retrospective analysis of the safety, efficacy, and feasibility data from 221 consecutive patients who underwent zero‐fluoroscopy ablation. Of these patients, 181 had atrial fibrillation, 17 had paroxysmal supraventricular tachycardia, 13 had atrial tachycardia, 6 had ventricular tachycardia, and 4 had ventricular premature contractions. We performed zero‐fluoroscopy ablation using three‐dimensional electro‐anatomical mapping systems and intracardiac echocardiography imaging. Ultrasound‐guided sheath insertion was performed on all cases. Our experience includes exclusively endocardial cardiac ablations. The mean follow‐up was 24 months. The recurrence rates were 25.4% for atrial fibrillation, 5.9% for paroxysmal supraventricular tachycardia, 15.4% for atrial tachycardia, 33.3% for ventricular tachycardia, and 25% for ventricular premature contraction. Complications occurred in two patients (0.9%), and there was no occurrence of death. A fluoroscopic guide was used in three cases for the confirmation of vascular access (one case) and for complications (two cases). Conclusions Zero‐fluoroscopy ablation was routinely performed without compromising on safety and efficacy. This approach may eliminate the exposure to radiation for all individuals involved in this procedure.

Environmental stress increases the risk of electron accumulation in photosystem I (PSI) of chloroplasts, which can cause oxygen (O2) reduction to superoxide radicals and decreased photosynthetic ability. We used three Arabidopsis thaliana lines: wild-type (WT) and the mutants pgr5hope1 and paa1-7/pox1. These lines have different reduced states of iron/sulfur (Fe/S) signals, including Fx, FA/FB, and ferredoxin, the electron carriers at the acceptor side of PSI. In the dark, short-pulse light was repetitively illuminated to the intact leaves of the plants to provide electrons to the acceptor side of PSI. WT and pgr5hope1 plants showed full reductions of Fe/S during short-pulse light and PSI inactivation. In contrast, paa1-7/pox1 showed less reduction of Fe/S and its PSI was not inactivated. Under continuous actinic-light illumination, pgr5hope1 showed no P700 oxidation with higher Fe/S reduction due to the loss of photosynthesis control and PSI inactivation. These results indicate that the accumulation of electrons at the acceptor side of PSI may trigger the production of superoxide radicals. P700 oxidation, responsible for the robustness of photosynthetic organisms, participates in reactive oxygen species suppression by oxidizing the acceptor side of PSI.

Against the potential risk in oxygenic photosynthesis, that is, the generation of reactive oxygen species, photosynthetic electron transport needs to be regulated in response to environmental fluctuations. One of the most important regulations is keeping the reaction center chlorophyll (P700) of photosystem I in its oxidized form in excess light conditions. The oxidation of P700 is supported by dissipating excess electrons safely to O2, and we previously found that the molecular mechanism of the alternative electron sink is changed from flavodiiron proteins (FLV) to photorespiration in the evolutionary history from cyanobacteria to plants. However, the overall picture of the regulation of photosynthetic electron transport is still not clear in bryophytes, the evolutionary intermediates. Here, we investigated the physiological roles of FLV and photorespiration for P700 oxidation in the liverwort Marchantia polymorpha by using the mutants deficient in FLV (flv1) at different O2 partial pressures. The effective quantum yield of photosystem II significantly decreased at 2kPa O2 in flv1, indicating that photorespiration functions as the electron sink. Nevertheless, it was clear from the phenotype of flv1 that FLV was dominant for P700 oxidation in M. polymorpha. These data suggested that photorespiration has yet not replaced FLV in functioning for P700 oxidation in the basal land plant probably because of the lower contribution to lumen acidification, compared with FLV, as reflected in the results of electrochromic shift analysis.

Photorespiration coupled with CO2 assimilation is thought to act as a defense system against photoinhibition caused by osmotic stress. In the present study, we examined whether such a mechanism is operative for the protection of photosystem I (PSI) in rice (Oryza sativa L.) including transgenic plants with decreased and increased Rubisco content (RBCS-antisense and RBCS-sense plants, respectively). All plants were hydroponically grown and moderate osmotic stress was imposed using hydroponic culture solutions containing poly(ethylene glycol) (PEG) at 16% or 20% (w/v) for 2 d. In wild-type plants, the rates of CO2 assimilation (A) were significantly decreased by the PEG treatment, whereas the photorespiration activity estimated from the rates of electron transport in photosystem II (PSII) and A were not affected. The maximal quantum efficiency of PSII (Fv/Fm) and the maximal activity of PSI (Pm) were also not affected. In RBCS-antisense plants, A and the estimated photorespiration activity were considerably lower than those in wild-type plants in the presence or absence of the PEG treatment. Pm and both Fv/Fm and Pm decreased in the 16% PEG-treated and 20% PEG-treated RBCS-antisense plants, respectively. Thus, the decrease in Rubisco content led to the photoinhibition of PSI and PSII, indicating the importance of photorespiration coupled with CO2 assimilation for the protection of PSI from moderate PEG-induced osmotic stress. It was also shown that PSI was more sensitive to osmotic stress than PSII. In the PEG-treated wild-type and RBCS-antisense plants, osmotic-stress responses of the photosynthetic electron transport reactions upstream of PSI led to the oxidation of P700, which is thought to prevent PSI from over-reduction. Although such a defense system operated, it was not sufficient for the protection of PSI in RBCS-antisense plants. In addition, there were no large differences in the parameters measured between wild-type and RBCS-sense plants, as overproduction of Rubisco did not increase photorespiration activity.

PSI photoinhibition is usually avoided through P700 oxidation. Without this protective mechanism, excess light represents a potentially lethal threat to plants. PGR5 is suggested to be a major component of cyclic electron transport around PSI and is important for P700 oxidation in angiosperms. The known Arabidopsis PGR5 deficient mutant, pgr5-1, is incapable of P700 oxidation regulation and has been used in numerous photosynthetic studies. However, here it was revealed that pgr5-1 was a double mutant with exaggerated PSI photoinhibition. pgr5-1 significantly reduced growth compared to the newly isolated PGR5 deficient mutant, pgr5hope1. The introduction of PGR5 into pgr5-1 restored P700 oxidation regulation, but remained a pale-green phenotype, indicating that pgr5-1 had additional mutations. Both pgr5-1 and pgr5hope1 tended to cause PSI photoinhibition by excess light, but pgr5-1 exhibited an enhanced reduction in PSI activity. Introducing AT2G17240, a candidate gene for the second mutation into pgr5-1 restored the pale-green phenotype and partially restored PSI activity. Furthermore, a deficient mutant of PGRL1 complexing with PGR5 significantly reduced PSI activity in the double-deficient mutant with AT2G17240. From these results, we concluded that AT2G17240, named PSI photoprotection 1 (PTP1), played a role in PSI photoprotection, especially in PGR5/PGRL1 deficient mutants.

In response to decreases in the assimilation efficiency of CO2, plants oxidize the reaction center chlorophyll (P700) of photosystem I (PSI) to suppress reactive oxygen species (ROS) production. In hydro-cultured sunflower leaves experiencing essential mineral deficiencies, we analyzed the following parameters that characterize PSI and PSII: (1) the reduction-oxidation states of P700 [Y(I), Y(NA), and Y(ND)]; (2) the relative electron flux in PSII [Y(II)]; (3) the reduction state of the primary electron acceptor in PSII, QA (1 − qL); and (4) the non-photochemical quenching of chlorophyll fluorescence (NPQ). Deficiency treatments for the minerals N, P, Mn, Mg, S, and Zn decreased Y(II) with an increase in the oxidized P700 [Y(ND)], while deficiencies for the minerals K, Fe, Ca, B, and Mo decreased Y(II) without an increase in Y(ND). During the induction of photosynthesis, the above parameters showed specific responses to each mineral. That is, we could diagnose the mineral deficiency and identify which mineral affected the photosynthesis parameters.

Upon exposure to environmental stress, the primary electron donor in photosystem I (PSI), P700, is oxidized to suppress the production of reactive oxygen species that could oxidatively inactivate the function of PSI. The illumination of rice leaves with actinic light induces intrinsic fluctuations in the opening and closing of stomata, causing the net CO2 assimilation rate to fluctuate. We examined the effects of these intrinsic fluctuations on electron transport reactions. Under atmospheric O2 conditions (21 kPa), the effective quantum yield of photosystem II (PSII) (Y(II)) remained relatively high while the net CO2 assimilation rate fluctuated, which indicates the function of alternative electron flow. By contrast, under low O2 conditions (2 kPa), Y(II) fluctuated. These results suggest that photorespiration primarily drove the alternative electron flow. Photorespiration maintained the oxidation level of ferredoxin (Fd) throughout the fluctuation of the net CO2 assimilation rate. Moreover, the relative activity of photorespiration was correlated with both the oxidation level of P700 and the magnitude of the proton gradient across the thylakoid membrane in 21 kPa O2 conditions. These results show that photorespiration oxidized P700 by stimulating the proton gradient formation when CO2 assimilation was suppressed by stomatal closure.

The oxidation of P700 in photosystem I (PSI) is a robust mechanism that suppresses the production of reactive oxygen species. We researched the contribution of photorespiration to the oxidation of P700 in wheat leaves. We analyzed the effects of changes in partial pressures of CO2 and O2 on photosynthetic parameters. The electron flux in photosynthetic linear electron flow (LEF) exhibited a positive linear relationship with an origin of zero against the dissipation rate (vH+) of electrochromic shift (ECS; ΔpH across thylakoid membrane), indicating that cyclic electron flow around PSI did not contribute to H+ usage in photosynthesis/photorespiration. The vH+ showed a positive linear relationship with an origin of zero against the H+ consumption rates in photosynthesis/photorespiration (JgH+). These two linear relationships show that the electron flow in LEF is very efficiently coupled with H+ usage in photosynthesis/photorespiration. Lowering the intercellular partial pressure of CO2 enhanced the oxidation of P700 with the suppression of LEF. Under photorespiratory conditions, the oxidation of P700 and the reduction of the plastoquinone pool were stimulated with a decrease in JgH+, compared to non-photorespiratory conditions. These results indicate that the reduction-induced suppression of electron flow (RISE) suppresses the reduction of oxidized P700 in PSI under photorespiratory conditions. Furthermore, under photorespiratory conditions, ECS was larger and H+ conductance was lower against JgH+ than those under non-photorespiratory conditions. These results indicate that photorespiration enhances RISE and ΔpH formation by lowering H+ conductance, both of which contribute to keeping P700 in a highly oxidized state.

Oxygen (O2)-evolving photosynthetic organisms oxidize the reaction center chlorophyll, P700, in photosystem I (PSI) to suppress the production of reactive oxygen species. The oxidation of P700 is accompanied by alternative electron flow in PSI (AEF-I), which is not required for photosynthetic linear electron flow (LEF). To characterize AEF-I, we compared the redox reactions of P700 and ferredoxin (Fd) during the induction of carbon dioxide (CO2) assimilation in wheat leaves, using dark-interval relaxation kinetics analysis. Switching on an actinic light (1000 μmol photons m−2 s−1) at ambient CO2 partial pressure of 40 Pa and ambient O2 partial pressure of 21 kPa gradually oxidized P700 (P700+) and enhanced the reduction rate of P700+ (vP700) and oxidation rate of reduced Fd (vFd). The vFd showed a positive linear relationship with an apparent photosynthetic quantum yield of PSII (Y[II]) originating at point zero; the redox turnover of Fd is regulated by LEF via CO2 assimilation and photorespiration. The vP700 also showed a positive linear relationship with Y(II), but the intercept was positive, not zero. That is, the electron flux in PSI included the electron flux in AEF-I in addition to that in LEF. This indicates that the oxidation of P700 induces AEF-I. We propose a possible mechanism underlying AEF-I and its physiological role in the mitigation of oxidative damage.