Search

Care is a practice and form of labour making human survival and flourishing possible. This Symposium explores the place and work of care within housing movements, asking how care operates as a politics, an ethics, and a set of practices through which tenants survive—and ultimately seek to transform—the structural violence of capitalist housing systems. Situated in US cities with abiding associations with Blackness and indigeneity, papers in the Symposium examine housing movements that take care as the starting point. As we discuss in this introduction to the Symposium, in such movements, care operates as connective tissue across households and modes of difference; challenges relations of racial capitalism and settler colonialism that underlie dominant understandings of who deserves and can demand care; and drives calls for public care and experiments with non-propertised forms of ownership. Housing systems are care infrastructures, making housing movements a vital place for care work.

OBJECTIVE: To develop evidence- and consensus-based clinical practice guidelines for management of high-tone pelvic floor dysfunction (HTPFD). High-tone pelvic floor dysfunction is a neuromuscular disorder of the pelvic floor characterized by non-relaxing pelvic floor muscles, resulting in lower urinary tract and defecatory symptoms, sexual dysfunction, and pelvic pain. Despite affecting 80% of women with chronic pelvic pain, there are no uniformly accepted guidelines to direct the management of these patients. METHODS: A Delphi method of consensus development was used, comprising three survey rounds administered anonymously via web-based platform (Qualtrics XM) to national experts in the field of HTPFD recruited through targeted invitation between September and December 2021. Eleven experts participated with backgrounds in urology, urogynecology, minimally invasive gynecology, and pelvic floor physical therapy (PFPT) participated. Panelists were asked to rate their agreement with rated evidence-based statements regarding HTPFD treatment. Statements reaching consensus were used to generate a consensus treatment algorithm. RESULTS: A total of 31 statements were reviewed by group members at the first Delphi round with 10 statements reaching consensus. 28 statements were reposed in the second round with 17 reaching consensus. The putative algorithm met clinical consensus in the third round. There was universal agreement for PFPT as first-line treatment for HTPFD. If satisfactory symptom improvement is reached with PFPT, the patient can be discharged with a home exercise program. If no improvement after PFPT, second-line options include trigger or tender point injections, vaginal muscle relaxants, and cognitive behavioral therapy, all of which can also be used in conjunction with PFPT. Onabotulinumtoxin A injections should be used as third line with symptom assessment after 2-4 weeks. There was universal agreement that sacral neuromodulation is fourth-line intervention.

Image retrieval, i.e., finding desired images given a reference image, inherently encompasses rich, multi-faceted search intents that are difficult to capture solely using image-based measures. Recent work leverages text instructions to allow users to more freely express their search intents. However, existing work primarily focuses on image pairs that are visually similar and/or can be characterized by a small set of pre-defined relations. The core thesis of this paper is that text instructions can enable retrieving images with richer relations beyond visual similarity. To show this, we introduce MagicLens, a series of self-supervised image retrieval models that support open-ended instructions. MagicLens is built on a key novel insight: image pairs that naturally occur on the same web pages contain a wide range of implicit relations (e.g., inside view of), and we can bring those implicit relations explicit by synthesizing instructions via large multimodal models (LMMs) and large language models (LLMs). Trained on 36.7M (query image, instruction, target image) triplets with rich semantic relations mined from the web, MagicLens achieves comparable or better results on eight benchmarks of various image retrieval tasks than prior state-of-the-art (SOTA) methods. Remarkably, it outperforms previous SOTA but with a 50X smaller model size on multiple benchmarks. Additional human analyses on a 1.4M-image unseen corpus further demonstrate the diversity of search intents supported by MagicLens., Comment: Work in progress

Following natural language instructions by executing actions in digital environments (e.g. web-browsers and REST APIs) is a challenging task for language model (LM) agents. Unfortunately, LM agents often fail to generalize to new environments without human demonstrations. This work presents BAGEL, a method for bootstrapping LM agents without human supervision. BAGEL converts a seed set of randomly explored trajectories or synthetic instructions, into demonstrations, via round-trips between two noisy LM components: an LM labeler which converts a trajectory into a synthetic instruction, and a zero-shot LM agent which maps the synthetic instruction into a refined trajectory. By performing these round-trips iteratively, BAGEL quickly converts the initial distribution of trajectories towards those that are well-described by natural language. We use BAGEL demonstrations to adapt a zero shot LM agent at test time via in-context learning over retrieved demonstrations, and find improvements of over 2-13% absolute on ToolQA and MiniWob++, with up to 13x reduction in execution failures.

This paper assesses trending AI foundation models, especially emerging computer vision foundation models and their performance in natural landscape feature segmentation. While the term foundation model has quickly garnered interest from the geospatial domain, its definition remains vague. Hence, this paper will first introduce AI foundation models and their defining characteristics. Built upon the tremendous success achieved by Large Language Models (LLMs) as the foundation models for language tasks, this paper discusses the challenges of building foundation models for geospatial artificial intelligence (GeoAI) vision tasks. To evaluate the performance of large AI vision models, especially Meta's Segment Anything Model (SAM), we implemented different instance segmentation pipelines that minimize the changes to SAM to leverage its power as a foundation model. A series of prompt strategies was developed to test SAM's performance regarding its theoretical upper bound of predictive accuracy, zero-shot performance, and domain adaptability through fine-tuning. The analysis used two permafrost feature datasets, ice-wedge polygons and retrogressive thaw slumps because (1) these landform features are more challenging to segment than manmade features due to their complicated formation mechanisms, diverse forms, and vague boundaries; (2) their presence and changes are important indicators for Arctic warming and climate change. The results show that although promising, SAM still has room for improvement to support AI-augmented terrain mapping. The spatial and domain generalizability of this finding is further validated using a more general dataset EuroCrop for agricultural field mapping. Finally, we discuss future research directions that strengthen SAM's applicability in challenging geospatial domains.

Moderate-sized large language models (LLMs) -- those with 7B or 13B parameters -- exhibit promising machine translation (MT) performance. However, even the top-performing 13B LLM-based translation models, like ALMA, does not match the performance of state-of-the-art conventional encoder-decoder translation models or larger-scale LLMs such as GPT-4. In this study, we bridge this performance gap. We first assess the shortcomings of supervised fine-tuning for LLMs in the MT task, emphasizing the quality issues present in the reference data, despite being human-generated. Then, in contrast to SFT which mimics reference translations, we introduce Contrastive Preference Optimization (CPO), a novel approach that trains models to avoid generating adequate but not perfect translations. Applying CPO to ALMA models with only 22K parallel sentences and 12M parameters yields significant improvements. The resulting model, called ALMA-R, can match or exceed the performance of the WMT competition winners and GPT-4 on WMT'21, WMT'22 and WMT'23 test datasets., Comment: Accepted at ICML 2024

A majority of language technologies are tailored for a small number of high-resource languages, while relatively many low-resource languages are neglected. One such group, Creole languages, have long been marginalized in academic study, though their speakers could benefit from machine translation (MT). These languages are predominantly used in much of Latin America, Africa and the Caribbean. We present the largest cumulative dataset to date for Creole language MT, including 14.5M unique Creole sentences with parallel translations -- 11.6M of which we release publicly, and the largest bitexts gathered to date for 41 languages -- the first ever for 21. In addition, we provide MT models supporting all 41 Creole languages in 172 translation directions. Given our diverse dataset, we produce a model for Creole language MT exposed to more genre diversity than ever before, which outperforms a genre-specific Creole MT model on its own benchmark for 26 of 34 translation directions., Comment: NAACL 2024

This paper focuses on the opportunities and the ethical and societal risks posed by advanced AI assistants. We define advanced AI assistants as artificial agents with natural language interfaces, whose function is to plan and execute sequences of actions on behalf of a user, across one or more domains, in line with the user's expectations. The paper starts by considering the technology itself, providing an overview of AI assistants, their technical foundations and potential range of applications. It then explores questions around AI value alignment, well-being, safety and malicious uses. Extending the circle of inquiry further, we next consider the relationship between advanced AI assistants and individual users in more detail, exploring topics such as manipulation and persuasion, anthropomorphism, appropriate relationships, trust and privacy. With this analysis in place, we consider the deployment of advanced assistants at a societal scale, focusing on cooperation, equity and access, misinformation, economic impact, the environment and how best to evaluate advanced AI assistants. Finally, we conclude by providing a range of recommendations for researchers, developers, policymakers and public stakeholders.

Recent generative AI systems have demonstrated more advanced persuasive capabilities and are increasingly permeating areas of life where they can influence decision-making. Generative AI presents a new risk profile of persuasion due the opportunity for reciprocal exchange and prolonged interactions. This has led to growing concerns about harms from AI persuasion and how they can be mitigated, highlighting the need for a systematic study of AI persuasion. The current definitions of AI persuasion are unclear and related harms are insufficiently studied. Existing harm mitigation approaches prioritise harms from the outcome of persuasion over harms from the process of persuasion. In this paper, we lay the groundwork for the systematic study of AI persuasion. We first put forward definitions of persuasive generative AI. We distinguish between rationally persuasive generative AI, which relies on providing relevant facts, sound reasoning, or other forms of trustworthy evidence, and manipulative generative AI, which relies on taking advantage of cognitive biases and heuristics or misrepresenting information. We also put forward a map of harms from AI persuasion, including definitions and examples of economic, physical, environmental, psychological, sociocultural, political, privacy, and autonomy harm. We then introduce a map of mechanisms that contribute to harmful persuasion. Lastly, we provide an overview of approaches that can be used to mitigate against process harms of persuasion, including prompt engineering for manipulation classification and red teaming. Future work will operationalise these mitigations and study the interaction between different types of mechanisms of persuasion.

What new progressive currents exist in the politics of housing? The ideas, projects, and policies profiled here promoted market-regulatory and redistributive agendas that can be termed progressive – stepping forward within, but pointing beyond, the neoliberal capitalist housing system. I compare multiple political processes in the housing system across the United States and Germany between 2008-2023, only one chapter being a direct comparison. Chapter one examines struggles over ideas, or how mass public discourse on social media illustrates policy under discussion before, during, and after peak-Covid-19. Chapter two analyzes struggles to collaboratively build new housing models, where people aspired to create broad participation, sustainable architecture, social spaces, in some cases decommodification. Chapter three directly compares struggles to create offensive policy change in Los Angeles and Berlin by tenant movement organizations and coalitions, to enact rent controls, expropriation of landlord property, and more. The cities provided representative examples of the radically inclusive and escalating influence of tenant power on policy in RALLY Cities (Renter, Activist, Large, Lefty). Finally, chapter four addresses struggles to take the policy offensive to the federal level, profiling a tenant network that created an innovative housing vision and built strategic partnerships with politicians, who advanced a new housing policy agenda in Washington DC. Since 2008 tenant organizations have mobilized offensively against the capitalist housing system. My findings suggest that shifting debate across the United States concentrated increasingly on policy tools that strongly intervene in markets and protect tenants, suggesting that trends in new housing politics may have implications beyond RALLY Cities. The case exploring the intricate path to US national housing reform also signals the potential for broader change. At scales local to national, tenants have built coali

Journal of Health promotion and maintenance with reports on systems-level public health interventions.

INTRODUCTION: The spectrum of bladder health and the factors that promote bladder health and prevent lower urinary tract symptoms (LUTS) among women are not well understood. This manuscript describes the rationale, aims, study design, sampling strategy, and data collection for the RISE FOR HEALTH (RISE) study, a novel study of bladder health in women conducted by the Prevention of Lower Urinary Tract Symptom (PLUS) Research Consortium. METHODS AND RESULTS: RISE is a population-based, multicenter, prospective longitudinal cohort study of community-dwelling, English- and Spanish-speaking adult women based in the United States. Its goal is to inform the distribution of bladder health and the individual factors (biologic, behavioral, and psychosocial) and multilevel factors (interpersonal, institutional, community, and societal) that promote bladder health and/or prevent LUTS in women across the life course. Key study development activities included the: (1) development of a conceptual framework and philosophy to guide subsequent activities, (2) creation of a study design and sampling strategy, prioritizing diversity, equity, and inclusion, and (3) selection and development of data collection components. Community members and cross-cultural experts shaped and ensured the appropriateness of all study procedures and materials. RISE participants will be selected by simple random sampling of individuals identified by a marketing database who reside in the 50 counties surrounding nine PLUS clinical research centers. Participants will complete self-administered surveys at baseline (mailed paper or electronic) to capture bladder health and LUTS, knowledge about bladder health, and factors hypothesized to promote bladder health and prevent LUTS. A subset of participants will complete an in-person assessment to augment data with objective measures including urogenital microbiome specimens. Initial longitudinal follow-up is planned at 1 year. DISCUSSION: Findings from RISE will b

OBJECTIVES: To describe the methods for the in-person musculoskeletal (MSK) assessment of the RISE FOR HEALTH (RISE) study, a population-based multicenter prospective cohort study designed to identify factors associated with bladder health (BH) conducted by the Prevention of Lower Urinary Tract Symptoms Research Consortium (PLUS). METHODS: A subset of RISE participants who express interest in the in-person assessment are screened to ensure eligibility (planned n = 525). Eligible consenting participants are asked to complete a standardized MSK assessment to evaluate core stability (four component core stability test, lumbar spine pain (seated slump test), pelvic girdle pain, (sacroiliac joint, anterior superior iliac spine, pubic symphysis tenderness, and pelvic girdle pain provocation test), hip pain (flexion, abduction, internal rotation and flexion, adduction and external rotation) and pelvic girdle function (active straight leg raise). Participants are also asked to complete the Short Physical Performance Battery to measure balance, gait speed, lower extremity strength, and functional capacity. RESULTS: Detailed online and in-person MSK training sessions led by physical therapy were used to certify research staff at each clinical center before the start of RISE in-person assessments. All evaluators exceeded the pre-specified pass rates. CONCLUSIONS: The RISE in-person MSK assessment will provide further insight into the role of general body MSK health and dysfunction and the spectrum of BH.

OBJECTIVES: To describe the methods for the in-person assessment of the RISE FOR HEALTH (RISE) study, a population-based multicenter prospective cohort study designed to identify factors that promote bladder health and/or prevent lower urinary tract symptoms in adult women, conducted by the Prevention of Lower Urinary Tract Symptoms Research Consortium (PLUS). METHODS AND RESULTS: A subset of RISE participants who express interest in the in-person assessment will be screened to ensure eligibility (planned n = 525). Eligible consenting participants are asked to complete 15 physical assessments in addition to height and weight, to assess pelvic floor muscle function, musculoskeletal (MSK) status, and pain, and to provide urogenital microbiome samples. Pelvic floor muscle assessments include presence of prolapse, strength, levator attachment integrity (tear) and myofascial pain. MSK tests evaluate core stability, lumbar spine, pelvic girdle and hip pain and function. Participants are asked to complete the Short Physical Performance Battery to measure balance, lower extremity strength, and functional capacity. All participants are asked to provide a voided urine sample and a vaginal swab for microbiome analyses; a subset of 100 are asked to contribute additional samples for feasibility and validation of a home collection of urinary, vaginal, and fecal biospecimens. RESULTS: Online and in-person training sessions were used to certify research staff at each clinical center before the start of RISE in-person assessments. Standardized protocols and data collection methods are employed uniformly across sites. CONCLUSIONS: The RISE in-person assessment is an integral portion of the overall population-based RISE study and represents an innovative approach to assessing factors hypothesized to promote bladder health and/or prevent lower urinary tract symptoms. Data collected from this assessment will be used to prioritize future research questions and prevention strategies and i

The relative time to maturity of grain crops is an important consideration for producers, yet there are no universally accepted classification schemes for cultivar phenology to guide decisions on variety selection and time of sowing. A first edition of an industry guide for wheat variety maturity was recently developed for use across Australia, representing a significant step forward for the grains industry. The aim of this paper was to revise and extend this industry guide to make it more robust, agronomically functional and meaningful to industry. The Australian Cereal Phenology Classification (ACPC) presented herein was developed using an unprecedented phenological data set with a diverse array of genotypes, environments and management. Field experiments were carried out with 70 wheat and 30 barley cultivars at 15 sites across Australia between 2017 and 2020. Thermal time to anthesis data were used to rank cultivars according to their relative phenology and divide them into classes, and then boundary cultivars of both species were selected to separate these classes. The resulting classification scheme divides wheat and barley into phenology classes ranging from ‘quick’ to ‘mid’ to ‘slow’. New cultivars to market can be assigned a phenology classification based on their thermal time to anthesis relative to the boundary cultivars. The ACPC will help growers, agronomists, breeders and researchers make informed decisions regarding cultivar comparison and selection while reducing misclassification and confusion across regions. The same methodology used to derive and validate the ACPC can be applied internationally to standardise descriptions of crop phenology.

Objective: The purpose of this analysis was to explore adolescent and adult women's interest in public health messaging around bladder health and perceptions of its usefulness. Materials and Methods: Directed content analysis of focus group data from the Study of Habits, Attitudes, Realities, and Experiences, which explored adolescent and adult women's experiences, perceptions, beliefs, knowledge, and behaviors related to bladder health across the life course. This article reports an analysis of the "Public Health Messaging" code, which included participants' desire or need for information about bladder health and recommendations for appropriate priority audiences. Results: Forty-four focus groups were conducted with 360 participants organized into six age groups (11-93 years). There was consensus across age groups that more information about the bladder is wanted and needed throughout the life course, as there is currently a lack of reliable educational resources. Information on bladder health was seen as useful and important because it enables people to anticipate negative changes in bladder health and act to prevent these. Several priority audiences were identified based on their risk of developing symptoms, but participants also saw value in educating the general public regardless of risk status. They also recommended education for parents and teachers who are in positions to control bathroom access. Conclusions: Results indicate a uniform desire for information on women's bladder health and a need for more research to develop individual prevention strategies and public health messaging for women of all ages, as well as guidance for organizations with a role in supporting bladder health.

TMEM106B, a lysosomal transmembrane protein, has been closely associated with brain health. Recently, an intriguing link between TMEM106B and brain inflammation has been discovered, but how TMEM106B regulates inflammation is unknown. Here, we report that TMEM106B deficiency in mice leads to reduced microglia proliferation and activation and increased microglial apoptosis in response to demyelination. We also found an increase in lysosomal pH and a decrease in lysosomal enzyme activities in TMEM106B-deficient microglia. Furthermore, TMEM106B loss results in a significant decrease in the protein levels of TREM2, an innate immune receptor essential for microglia survival and activation. Specific ablation of TMEM106B in microglia results in similar microglial phenotypes and myelination defects in mice, supporting the idea that microglial TMEM106B is critical for proper microglial activities and myelination. Moreover, the TMEM106B risk allele is associated with myelin loss and decreased microglial numbers in humans. Collectively, our study unveils a previously unknown role of TMEM106B in promoting microglial functionality during demyelination.

Heterozygous mutations in the granulin (GRN) gene, resulting in the haploinsufficiency of the progranulin (PGRN) protein, is a leading cause of frontotemporal lobar degeneration (FTLD). Complete loss of the PGRN protein causes neuronal ceroid lipofuscinosis (NCL), a lysosomal storage disorder. Polymorphisms in the GRN gene have also been associated with several other neurodegenerative diseases, including Alzheimers disease (AD), and Parkinsons disease (PD). PGRN deficiency has been shown to cause myelination defects previously, but how PGRN regulates myelination is unknown. Here, we report that PGRN deficiency leads to a sex-dependent myelination defect with male mice showing more severe demyelination in response to cuprizone treatment. This is accompanied by exacerbated microglial proliferation and activation in the male PGRN-deficient mice. Interestingly, both male and female PGRN-deficient mice show sustained microglial activation after cuprizone removal and a defect in remyelination. Specific ablation of PGRN in microglia results in similar sex-dependent phenotypes, confirming a microglial function of PGRN. Lipid droplets accumulate in microglia specifically in male PGRN-deficient mice. RNA-seq analysis and mitochondrial function assays reveal key differences in oxidative phosphorylation in male versus female microglia under PGRN deficiency. A significant decrease in myelination and accumulation of myelin debris and lipid droplets in microglia were found in the corpus callosum regions of FTLD patients with GRN mutations. Taken together, our data support that PGRN deficiency leads to sex-dependent alterations in microglia with subsequent myelination defects.

Large-scale multi-modal pre-training models such as CLIP and PaLI exhibit strong generalization on various visual domains and tasks. However, existing image classification benchmarks often evaluate recognition on a specific domain (e.g., outdoor images) or a specific task (e.g., classifying plant species), which falls short of evaluating whether pre-trained foundational models are universal visual recognizers. To address this, we formally present the task of Open-domain Visual Entity recognitioN (OVEN), where a model need to link an image onto a Wikipedia entity with respect to a text query. We construct OVEN-Wiki by re-purposing 14 existing datasets with all labels grounded onto one single label space: Wikipedia entities. OVEN challenges models to select among six million possible Wikipedia entities, making it a general visual recognition benchmark with the largest number of labels. Our study on state-of-the-art pre-trained models reveals large headroom in generalizing to the massive-scale label space. We show that a PaLI-based auto-regressive visual recognition model performs surprisingly well, even on Wikipedia entities that have never been seen during fine-tuning. We also find existing pretrained models yield different strengths: while PaLI-based models obtain higher overall performance, CLIP-based models are better at recognizing tail entities., Comment: Dataset available at https://open-vision-language.github.io/oven

Much of the previous work towards digital agents for graphical user interfaces (GUIs) has relied on text-based representations (derived from HTML or other structured data sources), which are not always readily available. These input representations have been often coupled with custom, task-specific action spaces. This paper focuses on creating agents that interact with the digital world using the same conceptual interface that humans commonly use -- via pixel-based screenshots and a generic action space corresponding to keyboard and mouse actions. Building upon recent progress in pixel-based pretraining, we show, for the first time, that it is possible for such agents to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based instruction following tasks.

We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. PaLI-X advances the state-of-the-art on most vision-and-language benchmarks considered (25+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.

Zero-shot cross-lingual transfer is when a multilingual model is trained to perform a task in one language and then is applied to another language. Although the zero-shot cross-lingual transfer approach has achieved success in various classification tasks, its performance on natural language generation tasks falls short in quality and sometimes outputs an incorrect language. In our study, we show that the fine-tuning process learns language invariant representations, which is beneficial for classification tasks but harmful for generation tasks. Motivated by this, we propose a simple method to regularize the model from learning language invariant representations and a method to select model checkpoints without a development set in the target language, both resulting in better generation quality. Experiments on three semantically diverse generation tasks show that our method reduces the accidental translation problem by 68% and improves the ROUGE-L score by 1.5 on average., Comment: Findings of ACL 2023

The mixed-size placement benchmarks described in this book chapter directly address several of the shortcomings in previously published suboptimality benchmarks. Two new sets of placement examples are constructed, one targeting the role of nonlocal nets in suboptimality, and another targeting the role of white space and large variations in module sizes. The first set, PEKO-MC, is a set of standard-cell circuits with nonlocal nets in known optimal placements. A given netlist is modified so as to render a given placement for it optimal for the new netlist. Cell dimensions and locations are not changed, net-degree statistics are matched exactly, and over 60% of the original netlist is left unchanged. The second set, PEKO-MS, incorporates a parametrized percentage of white space into a mixed-size placement which precisely matches given macro dimensions and locations as well as the net-degree distributions of the ISPD 2005 benchmark suite [21]. HPWL for the placements generated for the PEKO-MS circuits are proven to be less than 3% above optimal for most cases and within 8% of optimal on all cases., Comment: Version of Record: https://link.springer.com/book/10.1007/978-0-387-68739-1 Pre-print of Chapter 2 from the following work: Gi-Joon Nam and Jason Cong, Modern Circuit Placement Best Practices and Results, 2007, Springer, reproduced with permission of Springer Science+Business Media, LLC. The final authenticated version is available online at: http://dx.doi.org/10.1007/978-0-387-68739-1

Multilingual machine translation has proven immensely useful for both parameter efficiency and overall performance across many language pairs via complete multilingual parameter sharing. However, some language pairs in multilingual models can see worse performance than in bilingual models, especially in the one-to-many translation setting. Motivated by their empirical differences, we examine the geometric differences in representations from bilingual models versus those from one-to-many multilingual models. Specifically, we compute the isotropy of these representations using intrinsic dimensionality and IsoScore, in order to measure how the representations utilize the dimensions in their underlying vector space. Using the same evaluation data in both models, we find that for a given language pair, its multilingual model decoder representations are consistently less isotropic and occupy fewer dimensions than comparable bilingual model decoder representations. Additionally, we show that much of the anisotropy in multilingual decoder representations can be attributed to modeling language-specific information, therefore limiting remaining representational capacity., Comment: LREC-COLING 2024

Incorporating language-specific (LS) modules is a proven method to boost performance in multilingual machine translation. This approach bears similarity to Mixture-of-Experts (MoE) because it does not inflate FLOPs. However, the scalability of this approach to hundreds of languages (experts) tends to be unmanageable due to the prohibitive number of parameters introduced by full-rank matrices in fully-connected layers. In this work, we introduce the Language-Specific Matrix Synthesis (LMS) method. This approach constructs LS modules by generating low-rank matrices from two significantly smaller matrices to approximate the full-rank matrix. Furthermore, we condense multilingual knowledge from multiple LS modules into a single shared module with the Fuse Distillation (FD) technique to improve the efficiency of inference and model serialization. We show that our LMS method significantly outperforms previous LS methods and MoE methods with the same amount of extra parameters, e.g., 1.73 BLEU points over the Switch Transformer on many-to-many multilingual machine translation. Importantly, LMS is able to have comparable translation performance with much fewer parameters., Comment: Accepted at the main conference of EMNLP 2023

Internet links enable users to deepen their understanding of a topic by providing convenient access to related information. However, the majority of links are unanchored -- they link to a target webpage as a whole, and readers may expend considerable effort localizing the specific parts of the target webpage that enrich their understanding of the link's source context. To help readers effectively find information in linked webpages, we introduce the task of anchor prediction, where the goal is to identify the specific part of the linked target webpage that is most related to the source linking context. We release the AuthorAnchors dataset, a collection of 34K naturally-occurring anchored links, which reflect relevance judgments by the authors of the source article. To model reader relevance judgments, we annotate and release ReaderAnchors, an evaluation set of anchors that readers find useful. Our analysis shows that effective anchor prediction often requires jointly reasoning over lengthy source and target webpages to determine their implicit relations and identify parts of the target webpage that are related but not redundant. We benchmark a performant T5-based ranking approach to establish baseline performance on the task, finding ample room for improvement., Comment: 10 pages, 2 figures

Formulating selective information needs results in queries that implicitly specify set operations, such as intersection, union, and difference. For instance, one might search for "shorebirds that are not sandpipers" or "science-fiction films shot in England". To study the ability of retrieval systems to meet such information needs, we construct QUEST, a dataset of 3357 natural language queries with implicit set operations, that map to a set of entities corresponding to Wikipedia documents. The dataset challenges models to match multiple constraints mentioned in queries with corresponding evidence in documents and correctly perform various set operations. The dataset is constructed semi-automatically using Wikipedia category names. Queries are automatically composed from individual categories, then paraphrased and further validated for naturalness and fluency by crowdworkers. Crowdworkers also assess the relevance of entities based on their documents and highlight attribution of query constraints to spans of document text. We analyze several modern retrieval systems, finding that they often struggle on such queries. Queries involving negation and conjunction are particularly challenging and systems are further challenged with combinations of these operations., Comment: ACL 2023; Dataset available at https://github.com/google-research/language/tree/master/language/quest

Mixture-of-experts (MoE) models that employ sparse activation have demonstrated effectiveness in significantly increasing the number of parameters while maintaining low computational requirements per token. However, recent studies have established that MoE models are inherently parameter-inefficient as the improvement in performance diminishes with an increasing number of experts. We hypothesize this parameter inefficiency is a result of all experts having equal capacity, which may not adequately meet the varying complexity requirements of different tokens or tasks. In light of this, we propose Stratified Mixture of Experts (SMoE) models, which feature a stratified structure and can assign dynamic capacity to different tokens. We demonstrate the effectiveness of SMoE on three multilingual machine translation benchmarks, containing 4, 15, and 94 language pairs, respectively. We show that SMoE outperforms multiple state-of-the-art MoE models with the same or fewer parameters., Comment: Accepted at Findings of EMNLP 2023

We propose Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency. CoDA generalizes beyond standard adapter approaches to enable a new way of balancing speed and accuracy using conditional computation. Starting with an existing dense pretrained model, CoDA adds sparse activation together with a small number of new parameters and a light-weight training phase. Our experiments demonstrate that the CoDA approach provides an unexpectedly efficient way to transfer knowledge. Across a variety of language, vision, and speech tasks, CoDA achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approaches with moderate to no accuracy loss and the same parameter efficiency., Comment: NeurIPS camera ready version

We show that existing unsupervised methods on large language model (LLM) activations do not discover knowledge -- instead they seem to discover whatever feature of the activations is most prominent. The idea behind unsupervised knowledge elicitation is that knowledge satisfies a consistency structure, which can be used to discover knowledge. We first prove theoretically that arbitrary features (not just knowledge) satisfy the consistency structure of a particular leading unsupervised knowledge-elicitation method, contrast-consistent search (Burns et al. - arXiv:2212.03827). We then present a series of experiments showing settings in which unsupervised methods result in classifiers that do not predict knowledge, but instead predict a different prominent feature. We conclude that existing unsupervised methods for discovering latent knowledge are insufficient, and we contribute sanity checks to apply to evaluating future knowledge elicitation methods. Conceptually, we hypothesise that the identification issues explored here, e.g. distinguishing a model's knowledge from that of a simulated character's, will persist for future unsupervised methods., Comment: 12 pages (38 including references and appendices). First three authors equal contribution, randomised order

Large language models trained primarily in a monolingual setting have demonstrated their ability to generalize to machine translation using zero- and few-shot examples with in-context learning. However, even though zero-shot translations are relatively good, there remains a discernible gap comparing their performance with the few-shot setting. In this paper, we investigate the factors contributing to this gap and find that this gap can largely be closed (for about 70%) by matching the writing styles of the target corpus. Additionally, we explore potential approaches to enhance zero-shot baselines without the need for parallel demonstration examples, providing valuable insights into how these methods contribute to improving translation metrics.

Text generation models are notoriously vulnerable to errors in the training data. With the wide-spread availability of massive amounts of web-crawled data becoming more commonplace, how can we enhance the robustness of models trained on a massive amount of noisy web-crawled text? In our work, we propose Error Norm Truncation (ENT), a robust enhancement method to the standard training objective that truncates noisy data. Compared to methods that only uses the negative log-likelihood loss to estimate data quality, our method provides a more accurate estimation by considering the distribution of non-target tokens, which is often overlooked by previous work. Through comprehensive experiments across language modeling, machine translation, and text summarization, we show that equipping text generation models with ENT improves generation quality over standard training and previous soft and hard truncation methods. Furthermore, we show that our method improves the robustness of models against two of the most detrimental types of noise in machine translation, resulting in an increase of more than 2 BLEU points over the MLE baseline when up to 50% of noise is added to the data., Comment: ICLR 2024

An extensive library of symptom inventories has been developed over time to measure clinical symptoms, but this variety has led to several long standing issues. Most notably, results drawn from different settings and studies are not comparable, which limits reproducibility. Here, we present an artificial intelligence (AI) approach using semantic textual similarity (STS) to link symptoms and scores across previously incongruous symptom inventories. We tested the ability of four pre-trained STS models to screen thousands of symptom description pairs for related content - a challenging task typically requiring expert panels. Models were tasked to predict symptom severity across four different inventories for 6,607 participants drawn from 16 international data sources. The STS approach achieved 74.8% accuracy across five tasks, outperforming other models tested. This work suggests that incorporating contextual, semantic information can assist expert decision-making processes, yielding gains for both general and disease-specific clinical assessment.

One of the most surprising puzzles in neural network generalisation is grokking: a network with perfect training accuracy but poor generalisation will, upon further training, transition to perfect generalisation. We propose that grokking occurs when the task admits a generalising solution and a memorising solution, where the generalising solution is slower to learn but more efficient, producing larger logits with the same parameter norm. We hypothesise that memorising circuits become more inefficient with larger training datasets while generalising circuits do not, suggesting there is a critical dataset size at which memorisation and generalisation are equally efficient. We make and confirm four novel predictions about grokking, providing significant evidence in favour of our explanation. Most strikingly, we demonstrate two novel and surprising behaviours: ungrokking, in which a network regresses from perfect to low test accuracy, and semi-grokking, in which a network shows delayed generalisation to partial rather than perfect test accuracy.

The lack of quality labeled data is one of the main bottlenecks for training Deep Learning models. As the task increases in complexity, there is a higher penalty for overfitting and unstable learning. The typical paradigm employed today is Self-Supervised learning, where the model attempts to learn from a large corpus of unstructured and unlabeled data and then transfer that knowledge to the required task. Some notable examples of self-supervision in other modalities are BERT for Large Language Models, Wav2Vec for Speech Recognition, and the Masked AutoEncoder for Vision, which all utilize Transformers to solve a masked prediction task. GeoAI is uniquely poised to take advantage of the self-supervised methodology due to the decades of data collected, little of which is precisely and dependably annotated. Our goal is to extract building and road segmentations from Digital Elevation Models (DEM) that provide a detailed topography of the earths surface. The proposed architecture is the Masked Autoencoder pre-trained on ImageNet (with the limitation that there is a large domain discrepancy between ImageNet and DEM) with an UperNet Head for decoding segmentations. We tested this model with 450 and 50 training images only, utilizing roughly 5% and 0.5% of the original data respectively. On the building segmentation task, this model obtains an 82.1% Intersection over Union (IoU) with 450 Images and 69.1% IoU with only 50 images. On the more challenging road detection task the model obtains an 82.7% IoU with 450 images and 73.2% IoU with only 50 images. Any hand-labeled dataset made today about the earths surface will be immediately obsolete due to the constantly changing nature of the landscape. This motivates the clear necessity for data-efficient learners that can be used for a wide variety of downstream tasks.

To foster the development of new models for collaborative AI-assisted report generation, we introduce MegaWika, consisting of 13 million Wikipedia articles in 50 diverse languages, along with their 71 million referenced source materials. We process this dataset for a myriad of applications, going beyond the initial Wikipedia citation extraction and web scraping of content, including translating non-English articles for cross-lingual applications and providing FrameNet parses for automated semantic analysis. MegaWika is the largest resource for sentence-level report generation and the only report generation dataset that is multilingual. We manually analyze the quality of this resource through a semantically stratified sample. Finally, we provide baseline results and trained models for crucial steps in automated report generation: cross-lingual question answering and citation retrieval., Comment: Submitted to ACL, 2023