In industrial production, fabric products will always inevitably appear flaws due to uncontrollable factors such as production and transportation. However, there are many problems with the manual inspection methods used by manufacturers, such as low efficiency of fabric defects, high false detection rate, and high missed detection rate. While the diversity and complexity of fabric flaws also lead to the unsatisfactory results of existing flaw detection. Therefore, improving the detection and classification of fabric defects has become the key to problem solving. In this article, we propose a new deep convolutional network with attention mechanism (RDUnet-A) to solve the problems in fabric defect detection. The network is more efficient through training, and it is more helpful to realize the defect recognition of the image. We evaluated our model and the classic CNN model on the AITEX public data set, and the experimental results demonstrate that the newly proposed RDUnet-A model can achieve densely distributed defect detection, with Pixel Accuracy up to 0.600 and mlou up to 0.466, which is better than other classic models. This model effectively improves the accuracy and precision of fabric defect detection, and can obtain the defect location, which can meet industrial production needs basically.