Rotating dissolved air flotation (RDAF) has been utilized for several decades in paper‐recycling wastewater treatment; however, it has rarely been addressed in the literature, which makes research into this system challenging and complicated. However, in this work, a full‐scale industrial wastewater treatment system for a paper‐recycling mill in Mazandaran province, Iran, was evaluated. Experiments indicated that under the same wastewater and chemical conditions, there are differences in the removal efficiencies. This finding was investigated by conducting simulation in ANSYS CFX R18.0 and experimentation simultaneously. Thus, the main purpose of this research was to optimize the operation to achieve the highest removal efficiencies in a cost‐effective manner. For simulation, the flow was considered as a two‐phase homogeneous flow with a continuous solid–fluid phase, which discharges into the separation zone at different scenarios. Particles with diameters of 250, 500 and 1000 μm were introduced at four frame rotating speeds of 0.125, 0.25, 0.5 and 0.75 rpm. Measured qualitative parameters of the effluent included COD, BOD, TSS, TDS, conductivity and turbidity, whose removal efficiency ranges were (32.2%–54.8%), (32.7%–54.47%), (94.5%–97.1%), (0.47%–0.55%), (0.48%–0.59%) and (more than 55%), respectively. Moreover, the optimum rotating speeds of the frame for particles with diameters of 250, 500 and 1000 μm were obtained as 0.25, 0.5 and 0.125 rpm, respectively; these values were compared and found to agree with the experimental results. [ABSTRACT FROM AUTHOR]