Search

AimsGrowth differentiation factor-15 (GDF-15) and adiponectin are adipokines that regulate metabolism. This study aimed to evaluate the roles of GDF-15, adiponectin, and GDF-15/adiponectin ratio (G/A ratio) as biomarkers for detecting metabolic syndrome (MS).Materials and methodsThis cross-sectional study included 676 participants aged 20–70 years in Jurong, China. The participants were divided into four groups based on sex and age (ResultsThe prevalence of MS was 22.0% (149/676). Logistic regression analysis indicated that the G/A ratio and adiponectin levels, but not GDF-15 levels, were correlated with MS [odds ratio; 95% CI 1.010 (1.006–1.013) and 0.798 (0.735–0.865), respectively] after adjusting for confounding factors. The G/A ratio displayed a significant relationship with MS in each subgroup and with each MS component in both men and women; however, adiponectin concentrations were significantly associated with MS and all its components only in men (all P <0.05). The area under the curve (AUC) of the G/A ratio and the adiponectin level for MS was 0.758 and 0.748, respectively. The highest AUC was 0.757 for the adiponectin level in men and 0.724 for the G/A ratio in women.ConclusionsThis study suggests that the G/A ratio and adiponectin are potential biomarkers for detecting MS in women and men, respectively.

Recent years have seen an increase in the development of large deep learning (DL) models, which makes training efficiency crucial. Common practice is struggling with the trade-off between usability and performance. On one hand, DL frameworks such as PyTorch use dynamic graphs to facilitate model developers at a price of sub-optimal model training performance. On the other hand, practitioners propose various approaches to improving the training efficiency by sacrificing some of the flexibility, ranging from making the graph static for more thorough optimization (e.g., XLA) to customizing optimization towards large-scale distributed training (e.g., DeepSpeed and Megatron-LM). In this paper, we aim to address the tension between usability and training efficiency through separation of concerns. Inspired by DL compilers that decouple the platform-specific optimizations of a tensor-level operator from its arithmetic definition, this paper proposes a schedule language to decouple model execution from definition. Specifically, the schedule works on a PyTorch model and uses a set of schedule primitives to convert the model for common model training optimizations such as high-performance kernels, effective 3D parallelism, and efficient activation checkpointing. Compared to existing optimization solutions, we optimize the model as-needed through high-level primitives, and thus preserving programmability and debuggability for users to a large extent. Our evaluation results show that by scheduling the existing hand-crafted optimizations in a systematic way, we are able to improve training throughput by up to 3.35x on a single machine with 8 NVIDIA V100 GPUs, and by up to 1.32x on multiple machines with up to 64 GPUs, when compared to the out-of-the-box performance of DeepSpeed and Megatron-LM.

Supplemental Material for Carboxyl-functionalized nanocellulose/sulfonated poly(aryl ether ether ketone ketone) composites for proton exchange membrane by electrospinning by Xiaohui Guo, Xue Yang, Wanli Liu, Shuai Zheng, Yuanrui Wang, Wei Hu, Baijun Liu in High Performance Polymers

Additional file 1: Table S1. Primers used for real-time PCR.

Additional file 2: Supplemental Figure 1. The neutrophil cells characterized by CD11b+Ly6G+. Supplemental Figure 2. The levels of cytokines and chemokines in BALF of mice with administered 20mg/Kg APCP and 40mg/Kg APCP to wild type mice followed by A.f. Supplemental Figure 3. Eosinophil cells number and eosinophil chemokines expression in BALF. Supplemental Figure 4. The expression of other adenosine receptor subtypes. Supplemental Figure 5. The airway resistance of the WT and KO groups. Supplemental Figure 6. A2aR expression of wild-type mice and CD73 knockout mice. Supplemental Figure 7. Wild-type mice treated with adenosine, adenosine and A.f. induced inflammation.

This work proposes POMP, a prompt pre-training method for vision-language models. Being memory and computation efficient, POMP enables the learned prompt to condense semantic information for a rich set of visual concepts with over twenty-thousand classes. Once pre-trained, the prompt with a strong transferable ability can be directly plugged into a variety of visual recognition tasks including image classification, semantic segmentation, and object detection, to boost recognition performances in a zero-shot manner. Empirical evaluation shows that POMP achieves state-of-the-art performances on 21 downstream datasets, e.g., 67.0% average accuracy on 10 classification dataset (+3.1% compared to CoOp) and 84.4 hIoU on open-vocabulary Pascal VOC segmentation (+6.9 compared to ZSSeg)., Comment: Code is available at https://github.com/amazon-science/prompt-pretraining

Purpose: Bladder cancer (BCa) is a common malignancy in the urinary system. This study aims to explore the role of miR-186 in BCa tumorigenesis. Methods: The expression of miR-186 and ADAMTS12 in clinical BCa tissues and cell lines was detected. BCa cell lines T24, 5637 and EJ were used to transfect miR-186 mimics or inhibitors. Luciferase reporter gene detection confirmed the correlation between miR-186 and ADAMTS12. MTT method and flow cytometry were used to detect cell viability and apoptosis. Cell migration and invasion ability was detected by transwell assay. The protein level of ADAMTS12, β-catenin, GSK-3β and p-GSK-3β was determined using western blot analysis. Results: MiR-186 was negatively correlated with the expression of ADAMTS12 in BCa tissues. Further research confirmed that ADAMTS12 is the direct target of miR-186. In addition, overexpression of miR-186 down-regulated the expression of ADAMTS12,inhibiting cell viability and apoptosis, while knockout of miR-186 led to the opposite result. miR-186 also inhibits the phosphorylation of GSK-3 β and β-catenin without changing the total GSK-3β level. Our study shows that miR-186 has a negative regulatory effecton the expression of ADAMTS12 in clinical specimens and in vitro. miR-186 can inhibit the proliferation and invasion of BCa cells. Conclusions: miR-186 has the potential to be used as a biomarker in the early detection of BCa. Keywords: Purpose: Bladder cancer (BCa) is a common malignancy in the urinary system. This study aims to explore the role of miR-186 in BCatumorigenesis.Methods: The expression of miR-186 and ADAMTS12 in clinical BCa tissues and cell lines was detected. BCa cell lines T24, 5637 andEJ were used to transfect miR-186 mimics or inhibitors. Luciferase reporter gene detection confirmed the correlation between miR-186 and ADAMTS12. MTT method and flow cytometry were used to detect cell viability and apoptosis. Cell migration and invasionability was detected by transwell assay. The protein level of ADAMTS12, β-catenin, GSK-3β and p-GSK-3β was determined usingwestern blot analysis.Results: MiR-186 was negatively correlated with the expression of ADAMTS12 in BCa tissues. Further research confirmed thatADAMTS12 is the direct target of miR-186. In addition, overexpression of miR-186 down-regulated the expression of ADAMTS12,inhibiting cell viability and apoptosis, while knockout of miR-186 led to the opposite result. miR-186 also inhibits the phosphorylationof GSK-3 β and β-catenin without changing the total GSK-3β level. Our study shows that miR-186 has a negative regulatory effecton the expression of ADAMTS12 in clinical specimens and in vitro. miR-186 can inhibit the proliferation and invasion of BCa cells.Conclusions: miR-186 has the potential to be used as a biomarker in the early detection of BCa.Keywords: miR-186; ADAMTS12; bladder cancer; proliferation; metastasis biomarker.

Recent approaches have shown that training deep neural networks directly on large-scale image-text pair collections enables zero-shot transfer on various recognition tasks. One central issue is how this can be generalized to object detection, which involves the non-semantic task of localization as well as semantic task of classification. To solve this problem, we introduce a vision-language embedding alignment method that transfers the generalization capabilities of a pretrained model such as CLIP to an object detector like YOLOv5. We formulate a loss function that allows us to align the image and text embeddings from the pretrained model CLIP with the modified semantic prediction head from the detector. With this method, we are able to train an object detector that achieves state-of-the-art performance on the COCO, ILSVRC, and Visual Genome zero-shot detection benchmarks. During inference, our model can be adapted to detect any number of object classes without additional training. We also find that standard object detection scaling can transfer well to our method and find consistent improvements across various scales of YOLOv5 models and the YOLOv3 model. Lastly, we develop a self-labeling method that provides a significant score improvement without needing extra images nor labels., Comment: Code: https://github.com/Johnathan-Xie/ZSD-YOLO

We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B, their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system. Though we train using 70% spoken-form data, our teacher models perform comparably to XLM-R and mT5 when evaluated on the written-form Cross-lingual Natural Language Inference (XNLI) corpus. We perform a second stage of pretraining on our teacher models using in-domain data from our system, improving error rates by 3.86% relative for intent classification and 7.01% relative for slot filling. We find that even a 170M-parameter model distilled from our Stage 2 teacher model has 2.88% better intent classification and 7.69% better slot filling error rates when compared to the 2.3B-parameter teacher trained only on public data (Stage 1), emphasizing the importance of in-domain data for pretraining. When evaluated offline using labeled NLU data, our 17M-parameter Stage 2 distilled model outperforms both XLM-R Base (85M params) and DistillBERT (42M params) by 4.23% to 6.14%, respectively. Finally, we present results from a full virtual assistant experimentation platform, where we find that models trained using our pretraining and distillation pipeline outperform models distilled from 85M-parameter teachers by 3.74%-4.91% on an automatic measurement of full-system user dissatisfaction., Comment: KDD 2022

Traffic engineers are making efforts to mitigate congestion and to improve efficiency at intersections. The symmetric intersection (SI) can increase the capacity of intersections and is economical, practical, efficient and convenient. The pedestrians’ crossing patterns tailored for SI has not yet been studied. This paper not only develops three pedestrian crossing patterns but also analyzes crossing patterns from the aspects of efficiency and safety. For efficiency, delay models are proposed by considering through and diagonal pedestrian movements. For safety, exposure conflicts and the number of potential traffic accidents are analyzed. Then delay and potential accidents are converted into money value respectively and the total cost is calculated. The case study results show that SI performs better in terms of increasing capacity and decreasing pedestrian delay compared with conventional intersection. Crossing pattern 1 is efficient but unsafe while the total delay and safety cost of pattern 3 is usually the highest. The sensitivity analysis indicates that cycle length has a negative impact on average pedestrian cost and if critical flow ratio increases, the average pedestrian cost of pattern 1 and pattern 3 will increase and decrease respectively. Furthermore, the best pedestrian pattern choice under different values of cost per accident and cost of pedestrian delay is also studied.

Radiofrequency ablation (RFA) is a relatively new and effective therapeutic strategy for treating lung squamous cell carcinomas (LSCCs). However, RFA is rarely used in the clinic for LSCC which still suffers from a lack of effective comprehensive treatment strategies. In the present work, we investigate iDNMT, a novel small molecular inhibitor of DNMT1 with a unique structure. In clinical LSCC specimens, endogenous DNMT1 was positively associated with methylation rates of miR-27-3p′s promoter. Moreover, endogenous DNMT1 was negatively correlated with miR-27-3p expression which targets PSEN-1, the catalytic subunit of γ-secretase, which mediates the cleavage and activation of the Notch pathway. We found that DNMT1 increased activation of the Notch pathway in clinical LSCC samples while downregulating miR-27-3p expression and hypermethylation of miR-27-3p′s promoter. In addition of inhibiting activation of the Notch pathway by repressing methylation of the miR-27-3p promoter, treatment of LSCC cells with iDNMT1 also enhanced the sensitivity of LSCC tumor tissues to RFA treatment. These data suggest that iDNMT-induced inhibition of DNMT-1 enhances miR-27-3p expression in LSCC to inhibit activation of the Notch pathway. Furthermore, the combination of iDNMT and RFA may be a promising therapeutic strategy for LSCC.

The 18O/16O (and 15N/14N) ratio of natural nitrate (NO3-) and nitrite (NO2-) can be used to extract valuable information about their source and fate as environmental contaminants, their metabolism as macronutrients in plants and animals, and their behavior in the N biogeochemical cycle. We developed an accurate, precise, sensitive (minimum sample size: 0.2 μg NO3--equivalent), and reliable (minimal oxygen exchange, loss, or gain) method to selectively isolate and purify nitrate and nitrite from natural water, soil, air, and plant materials by strong anion exchange (SAX) for low- to normal-salinity samples or strong cation exchange (SCX) for high-salinity samples, followed by quantitative conversion to their respective benzyl esters, which can be separated and individually analyzed for δ18O (and potentially δ15N) by gas chromatography (GC)/pyrolysis/GC/isotope-ratio mass spectrometry (IRMS). The method compares favorably with the currently popular bacterial denitrification and chemical reduction methods, in terms of sensitivity and reliability, and has the potential to simultaneously measure δ15N and δ18O of nitrate and nitrite from natural samples of various origins.

Background Spinal cord injury (SCI) is a global medical problem. The smallest membrane-bound nanovesicles, known as exosomes, have a role in complex intercellular communication systems and can be used directly as therapeutic agents by acting as important paracrine factors. Nevertheless, the use of exosomes derived from BMSCs (BMSC-Exos) to treat SCI has been less, and the specific mechanism has not yet been reported. Methods BMSC-Exos were characterized by TEM, NTA and Western blot. The effects of BMSC-Exos treatment were compared by SCI in vivo model and a series of in vitro experiments. Results BMSC-Exos were found to enhance the expression of autophagy-related proteins LC3IIB and Beclin-1 and enabled autophagosomes formation. After BMSC-Exos treatment, there was marked decline in the level of expression of proapoptotic protein cleaved caspase-3, while that of the antiapoptotic protein Bcl-2 was upregulated. Conclusion BMSC-Exos can attenuate neuronal apoptosis by promoting autophagy and promote the potential efficacy of functional behavior recovery in SCI rats. In summary, these findings expand the theoretical knowledge and forms a realistic route for the future treatment of SCI by BMSC-Exos.

Supplemental material, sj-pdf-1-imr-10.1177_03000605221094526 for Clinical effects of prophylactic transverse colostomy in patients undergoing completely laparoscopic transabdominal approach partial intersphincteric resection by Honghong Zheng, Zhehong Li, Rui Su, Jianjun Li, Shuai Zheng, Ji Yang and Enhong Zhao in Journal of International Medical Research

The mixture of Expert (MoE) parallelism is a recent advancement that scales up the model size with constant computational cost. MoE selects different sets of parameters (i.e., experts) for each incoming token, resulting in a sparsely-activated model. Despite several successful applications of MoE, its training efficiency degrades significantly as the number of experts increases. The routing stage in MoE relies on the efficiency of the All2All communication collective, which suffers from network congestion and has poor scalability. To mitigate these issues, we introduce SMILE, which exploits heterogeneous network bandwidth and splits a single-step routing into bi-level routing. Our experimental results show that the proposed method obtains a 2.5x speedup over Switch Transformer in terms of pretraining throughput on the Colossal Clean Crawled Corpus without losing any convergence speed.

Existing out-of-distribution (OOD) detection methods are typically benchmarked on training sets with balanced class distributions. However, in real-world applications, it is common for the training sets to have long-tailed distributions. In this work, we first demonstrate that existing OOD detection methods commonly suffer from significant performance degradation when the training set is long-tail distributed. Through analysis, we posit that this is because the models struggle to distinguish the minority tail-class in-distribution samples, from the true OOD samples, making the tail classes more prone to be falsely detected as OOD. To solve this problem, we propose Partial and Asymmetric Supervised Contrastive Learning (PASCL), which explicitly encourages the model to distinguish between tail-class in-distribution samples and OOD samples. To further boost in-distribution classification accuracy, we propose Auxiliary Branch Finetuning, which uses two separate branches of BN and classification layers for anomaly detection and in-distribution classification, respectively. The intuition is that in-distribution and OOD anomaly data have different underlying distributions. Our method outperforms previous state-of-the-art method by $1.29\%$, $1.45\%$, $0.69\%$ anomaly detection false positive rate (FPR) and $3.24\%$, $4.06\%$, $7.89\%$ in-distribution classification accuracy on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT, respectively. Code and pre-trained models are available at https://github.com/amazon-research/long-tailed-ood-detection., Comment: ICML 2022

Adversarial training (AT) defends deep neural networks against adversarial attacks. One challenge that limits its practical application is the performance degradation on clean samples. A major bottleneck identified by previous works is the widely used batch normalization (BN), which struggles to model the different statistics of clean and adversarial training samples in AT. Although the dominant approach is to extend BN to capture this mixture of distribution, we propose to completely eliminate this bottleneck by removing all BN layers in AT. Our normalizer-free robust training (NoFrost) method extends recent advances in normalizer-free networks to AT for its unexplored advantage on handling the mixture distribution challenge. We show that NoFrost achieves adversarial robustness with only a minor sacrifice on clean sample accuracy. On ImageNet with ResNet50, NoFrost achieves $74.06\%$ clean accuracy, which drops merely $2.00\%$ from standard training. In contrast, BN-based AT obtains $59.28\%$ clean accuracy, suffering a significant $16.78\%$ drop from standard training. In addition, NoFrost achieves a $23.56\%$ adversarial robustness against PGD attack, which improves the $13.57\%$ robustness in BN-based AT. We observe better model smoothness and larger decision margins from NoFrost, which make the models less sensitive to input perturbations and thus more robust. Moreover, when incorporating more data augmentations into NoFrost, it achieves comprehensive robustness against multiple distribution shifts. Code and pre-trained models are public at https://github.com/amazon-research/normalizer-free-robust-training., Comment: ICML 2022

Supplemental Material for Analysis and Prediction of the Survival Trends of Patients with Clear-Cell Renal Cell Carcinoma: A Model-Based Period Analysis, 2001-2015 by Sicong Du, BD, Yu Zhong, MD, Shuai Zheng, MD, and Jun Lyu, PhD in Cancer Control.

Existing general purpose frameworks for gigantic model training, i.e., dense models with billions of parameters, cannot scale efficiently on cloud environment with various networking conditions due to large communication overheads. In this paper, we propose MiCS, which Minimizes the Communication Scale to bring down communication overhead. Specifically, by decreasing the number of participants in a communication collective, MiCS can utilize heterogeneous network bandwidth, reduce network traffic over slower links, reduce the latency of communications for maintaining high network bandwidth utilization, and amortize expensive global gradient synchronization overhead. Our evaluation on AWS shows that the system throughput of MiCS is up to 2.89× that of the state-of-the-art large model training systems. MiCS achieves near-linear scaling efficiency, which is up to 1.27× that of DeepSpeed. MiCS allows us to train a proprietary model with 100 billion parameters on 512 GPUs with 99.4% weak-scaling efficiency, and it is able to saturate over 54.5% theoretical computation power of each GPU on a public cloud with less GPU memory and more restricted networks than DGX-A100 clusters.

Graph neural networks (GNNs) have shown remarkable performance on homophilic graph data while being far less impressive when handling non-homophilic graph data due to the inherent low-pass filtering property of GNNs. In the face of analyzing complex real-world graphs with different homophily properties, the latent mixed local structural patterns in graphs should not be neglected. Therefore, the two questions, i.e., (\textbf{Q1}) and (\textbf{Q2}) as motioned above, should be well considered on the way to implementing a more generic GNN. For this purpose, we attempt to get deeper insights into them from two points, respectively, \textbf{(A1): Randomness of local patterns}, and \textbf{(A2): Aggregability of near-neighbors}., Comment: 3 pages, 1 figures

Supplemental Material for Recent Estimates and Predictions of 5-Year Survival in Patients with Gastric Cancer: A Model-Based Period Analysis by Yunmei Li, Aozi Feng, Shuai Zheng, Chong Chen, and Jun Lyu in Cancer Control

Additional file 1. TRIPOD checklist is provided for the prediction model development and validation.

ObjectiveTumor metabolism has always been the focus of cancer research. SLC16A1, as a key factor in catalysis of monocarboxylate transport across the plasma membrane, has been found to be associated with the occurrence and metastasis of a variety of cancers, but its prognostic significance and mechanism in different tumors are still unclear.MethodsBased on the gene expression matrix and clinical information of human cancer tissues acquired from TCGA and GTEX databases, the differential expression of SLC16A1 in different tumors and normal tissues was analyzed. To confirm the association between its expression, the mutation of MMRS gene, and the expression level of DNMTs. Univariate Cox regression was applied to analyze the association between SLC16A1 expression and patient prognosis. The effect of SLC16A1 expression on patient survival was examined by Kaplan Meier analysis. GSEA was used to identify related signaling pathways.ResultsThe expression of SLC16A1 was differentially expressed in most tumors, especially in the urinary tract where it is commonly highly expressed, and differential expression of SLC16A1 in different clinical stages. SLC16A1 expression was significantly positively correlated with MMRS gene mutation and DNMTS expression. Moreover, high SLC16A1 expression was associated with poorer overall survival (OS) and progression-free survival (PFS) in urological cancers. In particular, the results of the enrichment analysis showed that SLC16A1 was associated with processes such as cell adhesion and many signaling pathways affecting cell cycle were significantly enriched in the group with high-expressed SLC16A1.ConclusionSLC16A1 expression was upregulated in urological cancer. SLC16A1 may promote tumor development by regulating the epigenetic process of urological cancer and demonstrated a great potential as a prognostic biomarker of urological cancer patients.

Subject of the study: REIT (Real Estate Investment Trust) as equity investment instrument issued by management companies that are involved in the purchase, maintenance and construction of new real estate, and can also buy mortgage-backed securities from banks.The object of the study is the analysis (Real Estate Investment Trust) real estate market in China.The purpose of the work is to draw a conclusion about the attractiveness of investment in real estate in modern conditions relative to other tools and to identify the development prospects of REITs in ChinaResearch methods: descriptive-analytical, comparative-comparative, statistical, formant, contextual analysis methods.This thesis finds that equity REITs are still very immature and are not widely accepted by mass investors in China. The thesis puts forward three suggestions for apartment REITs in China: to streamline the REIT structure and avoid excessive complexity and opacity; to promote information transparency and regulated disclosure mechanism; to encourage and protect equity REIT investors by constraining REITs from taking on too much debt. Finally, the thesis concludes that REITs, as shown by US experience, should be regarded as a great opportunity to cultivate Chinese investors’ confidence in the stock market with its very simple and plain-vanilla structure. With the Chinese government’s strong ambition to increase housing affordability and to deleverage the economy, we foresee continuous legislative breakthroughs and more systematic improvements in the REIT field.Novelty elements: the current apartment operators are using REITs as debt-financing channels instead of real equity financing. As a result, the leverage level of the real estate sector may not really be decreased at all, but rather increased. With regard to this, the overall leverage level of the emerging apartment REIT companies should be a key factor that must be monitored to embody REIT as an equity investment vehicle.This work is dedicated to solve a number of difficulties in relevant fields of economy.

Using anesthetic gel may not sufficiently exclude pain perception during and after cystoscopy in male patients. To evaluate the analgesic efficacy and safety of intramuscular parecoxib (40 mg) for outpatient-based rigid cystoscopy, we performed a prospective, randomized and controlled study. Consecutive male patients requiring diagnostic cystoscopy in our hospital were divided into group A (1% tetracaine gel, n=50) and group B (parecoxib, n=51) at random. Patients received intramuscular injections of either 2 mL sterile saline in group A or 40 mg parecoxib in group B 30 min before the procedure. Tetracaine gel was injected into the urethra 3 min before the procedure in group A, with patients receiving plain lubricant gel in group B at the same time. Cystoscopy-associated pain levels were evaluated using the Visual Analog Score (VAS) during the procedure. Post-procedure urethral pain and complications were recorded and analyzed. The results showed that male patients experienced significantly less pain in group B than in group A (2.70±1.36 vs. 3.56±1.74, P=0.008). The percentage of patients with dysuria pain was not significantly different between the two groups. In addition, 24 h after cystoscopy, the patients with no previous experience of cystoscopy were more likely to declare urethral pain (59.2% vs. 33.3%, P=0.012, relative risk=1.78). No difference was observed in analgesic-related complications between the two groups. We conclude that intramuscular injection of 40 mg parecoxib may improve comfort for male patients undergoing rigid cystoscopy.

All the data of numerical simulation results shown in the paper.

Communication overhead severely hinders the scalability of distributed machine learning systems. Recently, there has been a growing interest in using gradient compression to reduce the communication overhead of the distributed training. However, there is little understanding of applying gradient compression to adaptive gradient methods. Moreover, its performance benefits are often limited by the non-negligible compression overhead. In this paper, we first introduce a novel adaptive gradient method with gradient compression. We show that the proposed method has a convergence rate of $\mathcal{O}(1/\sqrt{T})$ for non-convex problems. In addition, we develop a scalable system called BytePS-Compress for two-way compression, where the gradients are compressed in both directions between workers and parameter servers. BytePS-Compress pipelines the compression and decompression on CPUs and achieves a high degree of parallelism. Empirical evaluations show that we improve the training time of ResNet50, VGG16, and BERT-base by 5.0%, 58.1%, 23.3%, respectively, without any accuracy loss with 25 Gb/s networking. Furthermore, for training the BERT models, we achieve a compression rate of 333x compared to the mixed-precision training.

Background: Recruitment of gene modifying bone marrow mesenchymal stem cells (BMSCs) has been considered an alternative to single-cell injection in articular cartilage repair. Purpose: This study aimed to investigate whether the effect of runt-related transcription factor 2(Runx2) overexpression bone marrow mesenchymal stem cells in vivo could improve the quality of repaired tissue of a knee cartilage defect in a rabbit model. Methods: Thirty-two New Zealand rabbits were randomly divided into four groups. The blank group (Con) did not receive anything, the model group (Mo) was administered saline, the simple stem cell group (MSCs) received MSCs injection, and the Runx2 transfection group (R-MSCs) received Runx2 overexpression MSCs injection. After adapting to the environment for a week, a 5 mm diameter cylindrical osteochondral defect was created in the center of the medial femoral condyle. Cell and saline injections were performed in the first and third weeks after surgery. The cartilage repair was evaluated by macroscopically and microscopically at 4 and 8 weeks. Results: Macroscopically, defects were filled and surfaces were smoother in the MSCs groups than in the Mo group at 4th week. Microscopically, the R-MSCs group showed coloration similar to surrounding normal articular cartilage tissue at 8 weeks in masson trichrome staining. The COL-II, SOX9, and Aggrecan mRNA expressions of MSCs were enhanced at 4 weeks compared with R-MSCs, then the expression reduced at 8 weeks, but was still higher than Mo group level (P Conclusions: The R-MSCs group showed cellular morphology and arrangement similar to surrounding normal articular cartilage tissue, and Runx2 overexpression of MSCs resulted in overall superior cartilage repair as compared with MSCs at 8 weeks.

BERT has recently attracted a lot of attention in natural language understanding (NLU) and achieved state-of-the-art results in various NLU tasks. However, its success requires large deep neural networks and huge amount of data, which result in long training time and impede development progress. Using stochastic gradient methods with large mini-batch has been advocated as an efficient tool to reduce the training time. Along this line of research, LAMB is a prominent example that reduces the training time of BERT from 3 days to 76 minutes on a TPUv3 Pod. In this paper, we propose an accelerated gradient method called LANS to improve the efficiency of using large mini-batches for training. As the learning rate is theoretically upper bounded by the inverse of the Lipschitz constant of the function, one cannot always reduce the number of optimization iterations by selecting a larger learning rate. In order to use larger mini-batch size without accuracy loss, we develop a new learning rate scheduler that overcomes the difficulty of using large learning rate. Using the proposed LANS method and the learning rate scheme, we scaled up the mini-batch sizes to 96K and 33K in phases 1 and 2 of BERT pretraining, respectively. It takes 54 minutes on 192 AWS EC2 P3dn.24xlarge instances to achieve a target F1 score of 90.5 or higher on SQuAD v1.1, achieving the fastest BERT training time in the cloud., Comment: Technical Report (not under reviewed in any venue)

Learning from imperfect data becomes an issue in many industrial applications after the research community has made profound progress in supervised learning from perfectly annotated datasets. The purpose of the Learning from Imperfect Data (LID) workshop is to inspire and facilitate the research in developing novel approaches that would harness the imperfect data and improve the data-efficiency during training. A massive amount of user-generated data nowadays available on multiple internet services. How to leverage those and improve the machine learning models is a high impact problem. We organize the challenges in conjunction with the workshop. The goal of these challenges is to find the state-of-the-art approaches in the weakly supervised learning setting for object detection, semantic segmentation, and scene parsing. There are three tracks in the challenge, i.e., weakly supervised semantic segmentation (Track 1), weakly supervised scene parsing (Track 2), and weakly supervised object localization (Track 3). In Track 1, based on ILSVRC DET, we provide pixel-level annotations of 15K images from 200 categories for evaluation. In Track 2, we provide point-based annotations for the training set of ADE20K. In Track 3, based on ILSVRC CLS-LOC, we provide pixel-level annotations of 44,271 images for evaluation. Besides, we further introduce a new evaluation metric proposed by \cite{zhang2020rethinking}, i.e., IoU curve, to measure the quality of the generated object localization maps. This technical report summarizes the highlights from the challenge. The challenge submission server and the leaderboard will continue to open for the researchers who are interested in it. More details regarding the challenge and the benchmarks are available at https://lidchallenge.github.io, Comment: Summary of the 2nd Learning from Imperfect Data Workshop in conjunction with CVPR 2020

ZnGa2O4 is an important cathodoluminescent material which exhibits good near-infrared long afterglow characteristics after being Cr-doped into Ga sites (ZGO: Cr). The band structure, crystal structure and optical properties of ZGO: Cr are investigated by using first-principles calculations based on density functional theory. It is found that, in the ZGO: Cr system, due to the presence of the Cr-O bond, the distortion of the local lattice goes up as the Cr3+ doping amount increases. Meanwhile, impurity levels appear near the Fermi level of the ZGO system, and the orbital controlling the conduction band bottom gradually moves from the Ga-4s orbital to the Cr-3d orbital. Consequently, the real optical band-gap of the ZGO: Cr system becomes narrower, and the absorption edge undergoes a red-shift. The density of states of all upward and downward atomic spins is distributed asymmetrically, indicating that ZGO: Cr exhibits a certain amount of spin polarization. Additionally, the static dielectric constant of the ZGO: Cr material system increases with the increase in the Cr3+ doping amount. According to the findings, adjustments can be made of the doping concentration and energy range in the optical applications of ZGO: Cr material system.

Background Spinal cord injury (SCI) is a global medical problem. The smallest membrane-bound nanovesicles, known as exosomes, have a role in complex intercellular communication systems and can be used directly as therapeutic agents by acting as important paracrine factors. Nevertheless, the use of exosomes derived from BMSCs (BMSC-Exos) to treat SCI has been less, and the specific mechanism has not yet been reported. Methods BMSC-Exos were characterized by TEM, NTA and Western blot. The effects of BMSC-Exos treatment were compared by SCI in vivo model and a series of in vitro experiments. Results BMSC-Exos were found to enhance the expression of autophagy-related proteins LC3IIB and Beclin-1 and enabled autophagosomes formation. After BMSC-Exos treatment, there was marked decline in the level of expression of proapoptotic protein cleaved caspase-3, while that of the antiapoptotic protein Bcl-2 was upregulated. Conclusion BMSC-Exos can attenuate neuronal apoptosis by promoting autophagy and promote the potential efficacy of functional behavior recovery in SCI rats. In summary, these findings expand the theoretical knowledge and forms a realistic route for the future treatment of SCI by BMSC-Exos.

Caveolin-1 (CAV1) has been identified to be up-regulated in many cancers, including clear cell renal cell carcinoma (ccRCC). However, its potential function is still unclear in ccRCC. In this study, we demonstrated that CAV1 was frequently overexpressed in renal cell carcinoma tissues and cells, and was significantly associated with various clinicopathological parameters. In addition, high CAV1 expression was associated with poor disease-free survival (DFS) rate and could serve as a useful diagnostic indicator in ccRCC patients with different clinicopathological stages. Functional experiments demonstrated that CAV1 knockdown inhibited cell migration and invasion, whereas overexpression of CAV1 promoted cell migration and invasion in ccRCC. Moreover, CAV1 expression was up-regulated in sunitinib-resistant renal cancer cell lines, and its overexpression promoted sunitinib resistance. In general, our results confirm that CAV1 plays an important role in the metastasis of kidney cancer and induces sunitinib resistance, so CAV1 function suppression may become a promising clinical treatment strategy during renal cell carcinoma metastasis and sunitinib resistance.

A novel strain, 1433T, was isolated from leaves of Chinese red pepper (Huajiao, Zanthoxylum bungeanum Maxim) collected from Gansu province in northwestern China, and was characterised by a polyphasic approach. Cells of strain 1433T were observed to be Gram-stain positive, aerobic, asporogenous, rod shaped, motile and to have peritrichous flagella. The strain was observed to grow at a range of temperatures and pH, 4–45 °C (optimum 28–32 °C) and 6.0–10.0 (optimum pH 6.0–7.0), respectively. Growth was found to occur in the presence of 0–7% (w/v) NaCl [optimum 0–3% (w/v)]. The G+C content of the genomic DNA was determined to be 41.9 mol% and the cell wall peptidoglycan found to contain meso-diaminopimelic acid. The predominant menaquinone was identified as MK-7 and the major polar lipids as diphosphatidylglycerol, phosphatidylethanolamine, phosphatidylglycerol, an unidentified polar lipid and three unidentified phospholipids. The major cellular fatty acids were identified as iso-C15:0 (31.6%), anteiso-C15:0 (26.9%) and iso-C14:0 (17.1%). Phylogenetic analysis based on 16S rRNA gene sequences showed that strain 1433T is a member of the genus Bacillus and is closely related to Bacillus aryabhattai DSM 21047T (99.4% sequence similarity) and Bacillus megaterium DSM 32T (99.2%). DNA–DNA relatedness of the novel strain 1433T with B. aryabhattai DSM 21047T and B. megaterium DSM 32T was 33.8 ± 2.8% and 28.9 ± 3.4%, respectively. On the basis of the polyphasic evidence presented, strain 1433T is considered to represent a novel species of the genus Bacillus, for which we propose the name Bacillus zanthoxyli sp. nov. The type strain is 1433T (= CCTCC AB 2016326T = KCTC33730T).

We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating). These toolkits provide state-of-the-art pre-trained models, training scripts, and training logs, to facilitate rapid prototyping and promote reproducible research. We also provide modular APIs with flexible building blocks to enable efficient customization. Leveraging the MXNet ecosystem, the deep learning models in GluonCV and GluonNLP can be deployed onto a variety of platforms with different programming languages. The Apache 2.0 license has been adopted by GluonCV and GluonNLP to allow for software distribution, modification, and usage.

Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote achieves a 32x reduction on communication cost. However, its convergence is based on unrealistic assumptions and can diverge in practice. In this paper, we propose a general distributed compressed SGD with Nesterov's momentum. We consider two-way compression, which compresses the gradients both to and from workers. Convergence analysis on nonconvex problems for general gradient compressors is provided. By partitioning the gradient into blocks, a blockwise compressor is introduced such that each gradient block is compressed and transmitted in 1-bit format with a scaling factor, leading to a nearly 32x reduction on communication. Experimental results show that the proposed method converges as fast as full-precision distributed momentum SGD and achieves the same testing accuracy. In particular, on distributed ResNet training with 7 workers on the ImageNet, the proposed algorithm achieves the same testing accuracy as momentum SGD using full-precision gradients, but with $46\%$ less wall clock time., Comment: NeurIPS 2019

Efficient dispatching rule in manufacturing industry is key to ensure product on-time delivery and minimum past-due and inventory cost. Manufacturing, especially in the developed world, is moving towards on-demand manufacturing meaning a high mix, low volume product mix. This requires efficient dispatching that can work in dynamic and stochastic environments, meaning it allows for quick response to new orders received and can work over a disparate set of shop floor settings. In this paper we address this problem of dispatching in manufacturing. Using reinforcement learning (RL), we propose a new design to formulate the shop floor state as a 2-D matrix, incorporate job slack time into state representation, and design lateness and tardiness rewards function for dispatching purpose. However, maintaining a separate RL model for each production line on a manufacturing shop floor is costly and often infeasible. To address this, we enhance our deep RL model with an approach for dispatching policy transfer. This increases policy generalization and saves time and cost for model training and data collection. Experiments show that: (1) our approach performs the best in terms of total discounted reward and average lateness, tardiness, (2) the proposed policy transfer approach reduces training time and increases policy generalization., Comment: ECML PKDD 2019 (The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2019)

Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they can generalize worse than stochastic gradient descent. In this paper, by revisiting the design of Adagrad, we propose to split the network parameters into blocks, and use a blockwise adaptive stepsize. Intuitively, blockwise adaptivity is less aggressive than adaptivity to individual coordinates, and can have a better balance between adaptivity and generalization. We show theoretically that the proposed blockwise adaptive gradient descent has comparable convergence rate as its counterpart with coordinate-wise adaptive stepsize, but is faster up to some constant. We also study its uniform stability and show that blockwise adaptivity can lead to lower generalization error than coordinate-wise adaptivity. Experimental results show that blockwise adaptive gradient descent converges faster and improves generalization performance over Nesterov's accelerated gradient and Adam.

The aim of this study is to explore clonal growth in Populus pruinosa Schrenk and its role in the restoration and regeneration of desert riparian forests. From April to October 2014 and 2015, the survey of clonal growth was conducted in riparian forests along the Tarim River in Xinjiang, China. The study results showed that (1) From the aspect of space, the horizontal lateral roots distributed in 10–30 cm soil layer was the main body of P. pruinosa clone growth, and the adventitious buds, unearthed ramets and earthed ramets occurred and concentrated within 50 cm of clonal ramets. (2) From the aspect of time, no matter in the same year with different habitats, or in different year with same habitats, both the adventitious buds and unearthed ramets occurred concentrated in April–June, while earthed ramets occurred concentrated in May–October. (3) From the aspect of clonal ramets growth, conversion rates from adventitious buds to unearthed clonal ramets per unit length of horizontal lateral horizontal lateral roots were 41%, 5.0%, and 45%, and conversion rates from unearthed clonal ramets to earthed clonal ramets were 83.2%, 82.4% and 86% for edge (2014), internal (2014) woodland and edge (2015), respectively. The ramets growth rate and ramets population density in the unoccupied regions of woodland edge was more higher than that in internal woodland, at the end of growing season, the density of ramets population were up to 50 clusters/25 m 2 , plant height of clonal ramet up to 159.17 cm. (4) From April to July, soil moisture content was very significantly correlated with the number of adventitious buds and unearthed clone ramets. From August to October, the number of adventitious bud was significantly or very significantly correlated with land surface temperature and air temperature, the number of unearthed clonal ramets was very significantly correlated with the soil moisture content. In conclusion, in the process of P. pruinosa clonal growth, they adjust the number of ramets in each cluster and ramets density by changing number of adventitious bud occurrence and their conversion rate changed into unearthed ramets, and this adjustment process was affected by the seasonal variation of soil moisture content, land surface temperature and the temperature. clonal growth of P. pruinosa play an important role in the regeneration and restoration of natural woodland.

In order to quantitatively identify critical hidden damage for weld joints by using the metal magnetic memory technology (MMM), the modified maximum likelihood estimation MMM model is first proposed. The experimental materials are Q235B welded plate specimens. Fatigue tension experiments were operated to find the MMM feature laws of critical hidden damage by comparing with synchronous X-ray detection results. Four MMM characteristic parameters, that is, ΔHp(y) , Kymax , mmax and S(K), are extracted corresponding to the normal state and the hidden damage state, respectively. The probability density values of ΔHp(y) , Kymax , mmax and S(K)are calculated by the optimized bandwidth kernel density estimation. The quantitative maximum likelihood estimation MMM model is established based on optimized bandwidth kernel density. The verification result shows the maximum likelihood value of hidden damage state is twice as much as that of the normal state, which is consistent with the practical results. This provides a new method for quantitative MMM identification of weld critical hidden damages.

Silvered polyimide films, synthesized by an in-situ self-assembly method, find many applications in the fields of electronics and aerospace materials. In this work, polyimide/siloxanes were introduced into silvered PI films by an in-situ self-assembly process, which were resistant to atomic oxygen. Their atomic oxygen (AO) resistance behaviors were studied, wherein the mass loss of 20% Si-poly(amide acid salt) PAAS film was 30 ug cm−2 after exposure to AO fluence of 1.33 × 1020 O (10.5 h) and 190 ug cm−2 even after 42 h. Additionally, the volume resistivity value of the flexible PI/Ag composite film was 9.9 Ω cm−1 after 260 cycles of bending. The peel strength of PI/Ag/PI composite film was seven times more than that of the coated one (0.7 N cm-1 to 0.1 N cm-1). Moreover, (Dynamic Mechanical Analysis) DMA and (Thermogravimetric Analysis) TGA data also show that the introduction of self-encapsulating PAAS can not only increase the interlayer strength but maintain the performance of PI film. X-ray photoelectron spectroscopy, scanning electron microscopy, and atomic force microscopy were also employed to test the relative AO resistance behaviors. Thus, the silvered poly(siloxane amic acid) Si-PI composite films have AO resistance, high peel strength and alternating conductivity/ insulative effect, and hence can serve as a flexible printed circuit board for potential applications in low earth orbit.

Support Vector Machine (SVM) is an efficient classification approach, which finds a hyperplane to separate data from different classes. This hyperplane is determined by support vectors. In existing SVM formulations, the objective function uses L2 norm or L1 norm on slack variables. The number of support vectors is a measure of generalization errors. In this work, we propose a Minimal SVM, which uses L0.5 norm on slack variables. The result model further reduces the number of support vectors and increases the classification performance.

Singular value decomposition (SVD) is the mathematical basis of principal component analysis (PCA). Together, SVD and PCA are one of the most widely used mathematical formalism/decomposition in machine learning, data mining, pattern recognition, artificial intelligence, computer vision, signal processing, etc. In recent applications, regularization becomes an increasing trend. In this paper, we present a regularized SVD (RSVD), present an efficient computational algorithm, and provide several theoretical analysis. We show that although RSVD is non-convex, it has a closed-form global optimal solution. Finally, we apply RSVD to the application of recommender system and experimental result show that RSVD outperforms SVD significantly.

This chapter focuses on horizontal-axis wind turbine (HAWTs). It also focuses on a detailed analysis of the behavior of the fluid-structure interaction (FSIs) of wind turbines. In addition to the design theory, the chapter discusses the application and commercialization of wind turbines in relation to water desalination. The FSI of wind turbine blades will affect the power-generating efficiency, stability and lifespan of a wind turbine. The search for inexpensive, reliable and clean renewable energy sources for desalination of seawater and brackish water is a major challenge facing mankind. Wind turbines show great promise for providing electricity on a large scale, not only for desalination but also for applications such as heating and cooling in urban areas. A major long-term goal is to extensively deploy environmentally friendly and optimized wind turbine technology on a large scale to generate electricity to run water desalination plants.