Nonorthogonal multiple access (NOMA), which is an effective technique to achieve high spectral efficiency, has been emerging as a candidate for the fifth‐generation multiple access. In addition, NOMA reflects a transition on the forms of power allocation comparing with the orthogonal multiple access (OMA). In this paper, two relay selection schemes are considered to evaluate the achieved performance of a NOMA‐based relaying network, ie, the max‐min relay selection (MMRS) and the two‐stage relay selection (TSRS) schemes. Exact expression analyses for the system outage probability using the MMRS and TSRS schemes are derived to evaluate the system performance under Nakagami‐m fading. The existence of the error floor, caused by the estimation error, is confirmed by the detailed analysis of the asymptotic result. By comparing NOMA with OMA, and comparing the MMRS scheme with the TSRS scheme, one can reveal that cooperative NOMA under Nakagami‐m fading and imperfect channel estimation can maintain superior performance gain than OMA. A downlink NOMA‐based decode and forward relaying network is studied in this paper. Imperfect channel state information along with the Nakagami‐m fading are considered. Two relay selection schemes, ie, MMRS and TSRS schemes, are applied to analyze the system outage probability. [ABSTRACT FROM AUTHOR]