Agricultural non-point source pollution has become a matter of increasing public concern, and modern agriculture is gradually transforming into agglomeration, so it is important to study the influence of agricultural agglomeration on agricultural non-point source pollution to coordinate the relationship between resources, environment, and agricultural economic growth for guidance. With a focus on 89 prefecture-level cities in the main agricultural production areas of the Yangtze River basin in China, the authors analyzed the spatial and temporal evolution trends of agricultural agglomeration and agricultural non-point source pollution from 2000 to 2020 and then empirically tested the spatial spillover effects of agricultural agglomeration on agricultural non-point source pollution based on the spatial Durbin model (SDM). The results show that: (1) Between 2000 and 2020, agricultural agglomeration, in general, decreased from 0.364 to 0.342, and cities with agglomeration values in the third and fourth ranks are mainly located in the area north of the Yangtze River and have a tendency to extend southward over time. Agricultural non-point source pollution shows a general trend of increasing and then decreasing; its emissions rose from 404.319 × 104 tons in 2000 to 464.341 × 104 tons in 2015, and then fell to 373.338 × 104 tons in 2020, emissions in the third and fourth class of cities are mainly located in the middle and lower basin of the Yangtze River; High-value hots-pot areas of agricultural agglomeration, that is, areas with high spatial correlation, are mainly located in the upper and lower Yangtze River basin, and the areas with the higher spatial correlation of agricultural non-point source pollution are distributed in the upper, middle and lower basin of the Yangtze River. (2) The whole basin and upper basin exhibit positive and negative spatial spillover effects of agricultural non-point source pollution, Spillover effects values are 0.300 and −1.086, respectively; Agricultural agglomeration of the Whole Basin has a positive direct effect and a negative spatial spillover effect on agricultural non-point source pollution, the effect values are 0.846 and −0.520, respectively. (3) In addition to the core explanatory variable, agricultural production conditions and the share of livestock and poultry industry have a positive direct effect (the effect values are 0.109 and 0.048, respectively) and a negative spatial spillover effect (the effect values are −0.520 and −0.910, respectively) on agricultural non-point source pollution, while agricultural population size has a positive direct effect and spatial spillover effect, the effect values 0.099 and 0.452 respectively; The urbanization rate exacerbates the emission of agricultural non-point source pollution, the effect value is 0.110. while the industrial structure reduces the emission of agricultural non-point source pollution, the effect value is −0.438, but neither has a spatial spillover effect. The results imply that some effective policy measures, such as strengthening research on nutrient requirements and fertilization techniques for major crops, improving farmland infrastructure, scientifically planning and monitoring the scale of livestock farms, and strengthening inter-regional coordination and cooperation in the fight against pollution, should be taken to achieve ecological and sustainable objectives.