Background: Occupational characteristics are associated with late-life cognition. However, little is known about the association between occupational factors and cognition in early adulthood, especially when controlling for early life socioeconomic status (SES) and cognition in childhood. Importantly, sex may shape the impact of occupational characteristics that provide cognitive stimulation given that education, occupational status, and workplace experiences differ by sex. Methods: Using data on 12,129 participants ages 24–32 from the U.S.-based National Longitudinal Study of Adolescent to Adult Health, we investigated the association between four factors of occupational cognitive stimulation (repetition, freedom, analytic skills, and social interaction) and young-adult episodic and working memory independent of childhood and young-adult SES, using linear regression. We adjusted for confounding due to sex, race/ethnicity, age, childhood cognition, and education. We further investigated effect measure modification of this association by sex in stratified regression models. Results: Overall, 1-unit increases in both occupational analytic skills and social interaction were significantly associated with 0.101 (95%CI: 0.28, 0.173) and 0.096 (95%CI: 0.032, 0.160) SD higher memory, respectively. However, when sex-stratified, among men, a 1-unit increase on the social interaction scale was associated with 0.16 (95%CI: 0.05, 0.27) SD higher memory, while there was no association among women. Conclusion: Our results indicate that even in adulthood, activities that stimulate the mind can contribute to improved cognitive function, and the most beneficial forms of occupational stimulation are those that use analytic skills and involve social interaction (particularly among young men).