Objective: The objective of this study was to examine differences in worry, loneliness, and mental health between those individuals infected by COVID-19 or having someone their family infected, and the rest of the population. Methods: A cross-sectional online survey was conducted in Norway, UK, USA, and Australia during April/May 2020. Participants (n = 3810) were recruited via social media postings by the researchers and the involved universities. Differences between those with and without infection in the family were investigated with chi-square tests and independent t-tests. Multiple regression analyses were used to assess associations between sociodemographic variables and psychological outcomes (worry, loneliness, and mental health) in both groups. Results: Compared to their counterparts, participants with infection in the family reported higher levels of worries about themselves (p <, 0.05) and their family members (p <, 0.001) and had poorer mental health (p <, 0.05). However, the effect sizes related to the differences were small. The largest effect (d = 0.24) concerned worries about their immediate family. Poorer psychological outcomes were observed in those who were younger, female, unemployed, living alone and had lower levels of education, yet with small effect sizes. Conclusions: In view of the small differences between those with and without infection, we generally conclude that the mental health effects of the COVID-19 situation are not limited to those who have been infected or have had an infection within the family but extend to the wider population.