Search

StarCraft II (SC2) poses a grand challenge for reinforcement learning (RL), of which the main difficulties include huge state space, varying action space, and a long time horizon. In this work, we investigate a set of RL techniques for the full-length game of StarCraft II. We investigate a hierarchical RL approach involving extracted macro-actions and a hierarchical architecture of neural networks. We investigate a curriculum transfer training procedure and train the agent on a single machine with 4 GPUs and 48 CPU threads. On a 64x64 map and using restrictive units, we achieve a win rate of 99% against the level-1 built-in AI. Through the curriculum transfer learning algorithm and a mixture of combat models, we achieve a 93% win rate against the most difficult non-cheating level built-in AI (level-7). In this extended version of the paper, we improve our architecture to train the agent against the cheating level AIs and achieve the win rate against the level-8, level-9, and level-10 AIs as 96%, 97%, and 94%, respectively. Our codes are at https://github.com/liuruoze/HierNet-SC2. To provide a baseline referring the AlphaStar for our work as well as the research and open-source community, we reproduce a scaled-down version of it, mini-AlphaStar (mAS). The latest version of mAS is 1.07, which can be trained on the raw action space which has 564 actions. It is designed to run training on a single common machine, by making the hyper-parameters adjustable. We then compare our work with mAS using the same resources and show that our method is more effective. The codes of mini-AlphaStar are at https://github.com/liuruoze/mini-AlphaStar. We hope our study could shed some light on the future research of efficient reinforcement learning on SC2 and other large-scale games., Comment: 48 pages,21 figures

PR, IFPRI3; DCA; ISI, DSGD, The sustainable development of cooperatives is essential to agricultural sustainability and is determined mainly by cooperation longevity. The risk sharing and benefit distribution mechanisms are important governance arrangements that affect cooperation longevity in member-heterogeneous cooperatives. However, the relationship between cooperatives’ risk sharing and benefit distribution arrangements and cooperation longevity is not well understood. To address this research gap, we provide a conceptual framework to illustrate the relationship between risk sharing, benefit distribution, contract form and cooperation longevity. By conducting 23 interviews with three dairy cooperatives in China using a multiple-case-study method, we find that cooperation longevity is influenced by the alignment of the combination of risk sharing and benefit distribution for cooperatives and their members. Following the principles of risk-benefit equivalence and strategy-risk preference matching, we discuss four scenarios of match or mismatch of combinations of risk sharing and benefit distribution. We suggest that the government needs to strengthen risk mitigation capabilities for cooperatives and members, and to help them to build an effective risk sharing and benefit distribution arrangement to achieve sustainable development.

Jun Zhang,1– 3,* Chun-Lin Mai,1,* Ying Xiong,1 Zhen-Jia Lin,1 Ying-Tao Jie,1 Jie-Zhen Mai,1 Chong Liu,1 Man-Xiu Xie,4 Xin Zhou,2,3 Xian-Guo Liu1,5,6 1Department of Physiology and Pain Research Center, Zhongshan School of Medicine, Sun Yat-sen University, Guangzhou, 510080, People’s Republic of China; 2Department of Anesthesiology, Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences, Guangzhou, 510080, People’s Republic of China; 3Guangdong Cardiovascular Institute, Guangzhou, 510080, People’s Republic of China; 4Department of Anesthesiology, Sun Yat-sen University Cancer Center, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Guangzhou, 510060, People’s Republic of China; 5Department of Anesthesiology, Guangdong Second Provincial Central Hospital, Guangzhou, 510317, People’s Republic of China; 6Guangdong Province Key Laboratory of Brain Function and Disease, Guangzhou, 510080, People’s Republic of China*These authors contributed equally to this workCorrespondence: Xian-Guo Liu; Xin Zhou Email liuxg@mail.sysu.edu.cn; zhoux89@mail2.sysu.edu.cnPurpose: Postmenopausal women often suffer from chronic pain, memory decline and mood depression. The mechanisms underlying the neuronal disorders are not fully understood, and effective treatment is still lacking.Methods: Oral administration of magnesium-L-threonate was tested to treat the neuronal disorders in ovariectomized and aged female mice. The pain hypersensitivity, memory function and depression-like behaviors were measured with a set of behavioral tests. Western blots, immunochemistry and in situ hybridization were used to assess molecular changes.Results: Chronic oral administration of magnesium-L-threonate substantially prevented or reversed the chronic pain and memory/emotional deficits in both ovariectomized and aged female mice. We found that phospho-p65, an active form of nuclear factor-kappaB

Ying-Yi Luan,1,* Yan-Hong Chen,2,* Xue Li,3 Zhi-Peng Zhou,2 Jia-Jia Huang,2,4 Zhen-Jia Yang,2,4 Jing-Jing Zhang,2,5 Ming Wu2,4,6 1Department of Central Laboratory, Beijing Obstetrics and Gynecology Hospital, Capital Medical University, Beijing, 100026, People’s Republic of China; 2Department of Critical Care Medicine and Hospital Infection Prevention and Control, Shenzhen Second People`s Hospital & First Affiliated Hospital of Shenzhen University, Health Science Center, Shenzhen, 518035, People’s Republic of China; 3Department of Emergency, the Eighth Affiliated Hospital, Sun Yat-sen University, Shenzhen, 518033, People’s Republic of China; 4Shantou University Medical College, Shantou, 515041, People’s Republic of China; 5Department of Critical Care Medicine, Pingshan District People’s Hospital of Shenzhen, Shenzhen, 518118, People’s Republic of China; 6Guangxi University of Chinese Medicine, Nanning, 530200, People’s Republic of China*These authors contributed equally to this workCorrespondence: Ming Wu Tel +86 755 83676149Email boshiyy@126.comBackground: Increasing evidence indicates carbapenem-resistant Klebsiella pneumoniae (CrKP) is increasingly prevalent in intensive care unit (ICU), but its clinical characteristics and risk factors remain unknown.Aim: The aim of the present study was to evaluate clinical characteristics, risk factors in critically ill patients with CrKP infection.Methods: A retrospective study was included in patients from January 2013 to October 2019. Clinical data were collected from CrKP patients on the day of specimen collection admitted to ICU. Multivariable logistic regression was used for risk factors. Receiver operating curve (ROC) and the area under the curve (AUC) with DeLong method of MedCalc software were used. Two-way repeated-measures ANOVA analysis was used to analyze the characteristics of independent risk factors over time.Findings: A total of 147 adult patients with CrKP were screene

Learning-to-rank (LTR) has become a key technology in E-commerce applications. Most existing LTR approaches follow a supervised learning paradigm from offline labeled data collected from the online system. However, it has been noticed that previous LTR models can have a good validation performance over offline validation data but have a poor online performance, and vice versa, which implies a possible large inconsistency between the offline and online evaluation. We investigate and confirm in this paper that such inconsistency exists and can have a significant impact on AliExpress Search. Reasons for the inconsistency include the ignorance of item context during the learning, and the offline data set is insufficient for learning the context. Therefore, this paper proposes an evaluator-generator framework for LTR with item context. The framework consists of an evaluator that generalizes to evaluate recommendations involving the context, and a generator that maximizes the evaluator score by reinforcement learning, and a discriminator that ensures the generalization of the evaluator. Extensive experiments in simulation environments and AliExpress Search online system show that, firstly, the classic data-based metrics on the offline dataset can show significant inconsistency with online performance, and can even be misleading. Secondly, the proposed evaluator score is significantly more consistent with the online performance than common ranking metrics. Finally, as the consequence, our method achieves a significant improvement (\textgreater$2\%$) in terms of Conversion Rate (CR) over the industrial-level fine-tuned model in online A/B tests.

Injecting human knowledge is an effective way to accelerate reinforcement learning (RL). However, these methods are underexplored. This paper presents our discovery that an abstract forward model (thought-game (TG)) combined with transfer learning (TL) is an effective way. We take StarCraft II as our study environment. With the help of a designed TG, the agent can learn a 99% win-rate on a 64x64 map against the Level-7 built-in AI, using only 1.08 hours in a single commercial machine. We also show that the TG method is not as restrictive as it was thought to be. It can work with roughly designed TGs, and can also be useful when the environment changes. Comparing with previous model-based RL, we show TG is more effective. We also present a TG hypothesis that gives the influence of different fidelity levels of TG. For real games that have unequal state and action spaces, we proposed a novel XfrNet of which usefulness is validated while achieving a 90% win-rate against the cheating Level-10 AI. We argue that the TG method might shed light on further studies of efficient RL with human knowledge.

Understanding and accurately predicting within-field spatial variability of crop yield play a key role in site-specific management of crop inputs such as irrigation water and fertilizer for optimized crop production. However, such a task is challenged by the complex interaction between crop growth and environmental and managerial factors, such as climate, soil conditions, tillage, and irrigation. In this paper, we present a novel Spatial-temporal Multi-Task Learning algorithms for within-field crop yield prediction in west Texas from 2001 to 2003. This algorithm integrates multiple heterogeneous data sources to learn different features simultaneously, and to aggregate spatial-temporal features by introducing a weighted regularizer to the loss functions. Our comprehensive experimental results consistently outperform the results of other conventional methods, and suggest a promising approach, which improves the landscape of crop prediction research fields.

StarCraft II poses a grand challenge for reinforcement learning. The main difficulties of it include huge state and action space and a long-time horizon. In this paper, we investigate a hierarchical reinforcement learning approach for StarCraft II. The hierarchy involves two levels of abstraction. One is the macro-action automatically extracted from expert's trajectories, which reduces the action space in an order of magnitude yet remains effective. The other is a two-layer hierarchical architecture which is modular and easy to scale, enabling a curriculum transferring from simpler tasks to more complex tasks. The reinforcement training algorithm for this architecture is also investigated. On a 64x64 map and using restrictive units, we achieve a winning rate of more than 99\% against the difficulty level-1 built-in AI. Through the curriculum transfer learning algorithm and a mixture of combat model, we can achieve over 93\% winning rate of Protoss against the most difficult non-cheating built-in AI (level-7) of Terran, training within two days using a single machine with only 48 CPU cores and 8 K40 GPUs. It also shows strong generalization performance, when tested against never seen opponents including cheating levels built-in AI and all levels of Zerg and Protoss built-in AI. We hope this study could shed some light on the future research of large-scale reinforcement learning., Comment: Appeared in AAAI 2019

Context: Thyroglobulin (Tg) could be a sensitive biomarker of iodine nutrition in pregnant women (PW). A dried blood spot (DBS) assay would simplify collection and transport in field studies. Objectives: Our aims were to (1) establish and test a reference range for DBS-Tg in PW; (2) determine whether co-measurement of Tg antibodies (Abs) is necessary to define population iodine status. Design, Setting, and Participants: Standardized cross-sectional studies of 3870 PW from 11 countries. For the DBS-Tg reference range, we included TgAb-negative PW (n = 599) from 3 countries with sufficient iodine intake. Main Outcome Measures: We measured the urinary iodine concentration and DBS thyroid-stimulating hormone, total thyroxin, Tg, and TgAb. Results: In the reference population, the median DBS-Tg was 9.2 μg/L (95% confidence interval, 8.7 to 9.8 μg/L) and was not significantly different among trimesters. The reference range was 0.3 to 43.5 μg/L. Over a range of iodine intake, the Tg concentrations were U-shaped. Within countries, the median DBS-Tg and the presence of elevated DBS-Tg did not differ significantly between all PW and PW who were TgAb-negative. Conclusions: A median DBS-Tg of ∼10 μg/L with <3% of values ≥44 μg/L indicated population iodine sufficiency. Concurrent measurement of TgAb did not appear necessary to assess the population iodine status.

DBS-Tg is a sensitive biomarker of iodine status in PW, the reference range in iodine sufficient PW is 0.3-43.5 μg/L and a median DBS-Tg <10 μg/L indicates iodine sufficiency in PW.