Nutrient availability and plant diversity are two important factors determining crop productivity in agricultural ecosystems, but little is known about the underlying mechanisms shaping microbial communities and their regulatory roles in soil biological activity and function. Here, we explored the impacts of fertilization regimes and crop rotations on soil physicochemical properties, crop yield and bacterial and fungal community structures in a 26-year field experiment. The critical determinants for regulating soil enzyme activity profiles involved in carbon (C), nitrogen (N) and phosphorus (P) cycling were identified by the partial least squares path model (PLS-PM). Long-term inorganic or organic fertilization significantly increased soil total N by 27%–77% and crop yield by 237%–419% and decreased soil pH by an average of 0.4 units when compared with non-fertilized control. Soil bacteria were more sensitive than fungi to the fertilization practices. Nutrient additions enriched copiotrophic taxa affiliated to the Pseudomonadaceae and Cytophagaceae bacterial families, but reduced some Acidobacteria such as subgroup 4 RB41, which was the most sensitive biomarker responding to no fertilization. Conversely, fungi were more active in response to crop conversion from wheat-maize to wheat-soybean rotation, leading to a 3-fold enhancement of an unclassified Sordariomycetes family in soybean-based rotation. PLS-PM revealed that fertilization-induced increases in soil enzyme activities were regulated by the bacterial community, while plant-driven alterations in yield, organic C input and soil aggregate-size distribution played an important role for fungal development, which, however, had no significant link to soil enzyme activity profiles. Our results suggest that different response patterns of soil bacteria and fungi to agricultural practices might have consequences for ecosystem function.