Detection of rare, circulating tumor cells (CTC's) in human peripheral blood is a potential indicator of prognosis and diagnosis in oncology. Typical methods to detect these CTC's are either by immunocytochemistry (ICCS) or RT‐PCR. However without accurate, rapid, and reproducible enrichment processes, these detection techniques are labor intensive and/or unreliable. In this article, a repeatable enrichment process that included a flow‐through immunomagnetic cell separation system, the quadrupole magnetic sorter (QMS) was optimized with the aid of a statistical analysis software package. The QMS was operated in a negative mode of operation by immunomagnetically targeting normal human peripheral blood lymphocytes (PBL) through the CD45 surface marker. Three head and neck squamous carcinoma cell lines (HNSCC), Detroit‐562, SCC‐4, and CAL‐27, were used to determine the sensitivity of RT‐PCR for the epidermal growth factor receptor (EGFR) in spiked PBL. The detection purity needed for detection was found to be one cell in 104, one cell in 103, and one cell in 105 for the Detroit‐562, SCC‐4, and CAL‐27, respectively. The actual number of cancer cells needed for RT‐PCR detection ranged from 30 to 1 cell. To mimic the potential concentration of rare CTC present in peripheral blood of cancer patients, the spiking concentration was chosen to be one cancer cell per 105 total leukocytes from healthy donors. Using a single step immunomagnetic labeling, the final, optimized enrichment process produced a 57.6 ± 30.3‐fold (n = 6) enrichment of the rare cancer cells with a final cancer cell recovery of (77.8 ± 6.6)%. © 2007 Clinical Cytometry Society