Purpose: To compare the evaluation metrics for deep learning methods that were developed using imbalanced imaging data in osteoarthritis studies., Materials and Methods: This retrospective study utilized 2996 sagittal intermediate-weighted fat-suppressed knee MRIs with MRI Osteoarthritis Knee Score readings from 2467 participants in the Osteoarthritis Initiative study. We obtained probabilities of the presence of bone marrow lesions (BMLs) from MRIs in the testing dataset at the sub-region (15 sub-regions), compartment, and whole-knee levels based on the trained deep learning models. We compared different evaluation metrics (e.g., receiver operating characteristic (ROC) and precision-recall (PR) curves) in the testing dataset with various class ratios (presence of BMLs vs. absence of BMLs) at these three data levels to assess the model's performance., Results: In a subregion with an extremely high imbalance ratio, the model achieved a ROC-AUC of 0.84, a PR-AUC of 0.10, a sensitivity of 0, and a specificity of 1., Conclusion: The commonly used ROC curve is not sufficiently informative, especially in the case of imbalanced data. We provide the following practical suggestions based on our data analysis: 1) ROC-AUC is recommended for balanced data, 2) PR-AUC should be used for moderately imbalanced data (i.e., when the proportion of the minor class is above 5% and less than 50%), and 3) for severely imbalanced data (i.e., when the proportion of the minor class is below 5%), it is not practical to apply a deep learning model, even with the application of techniques addressing imbalanced data issues., Competing Interests: Declaration of Competing Interest S. L.: No relevant relationships. F. R.: Shareholders of BICL, LLC. Consultant to Calibr-California Institute of Biomedical Research and Grunenthal. Y. G.: No relevant relationships. E. J. B.: No relevant relationships. Z.-M. L.: No relevant relationships. A. G.: Consultant to Pfizer, Novartis, Regeneron, TissueGene, Merck Serono, and AstraZeneca. Shareholders of BICL, LLC. L. S.: No relevant relationships. C. E.: No relevant relationships. M.C.H.: Perform consulting activities, including attendance at virtual Advisory Board meetings, for the following entities: Acadia Pharmaceuticals, Bioclinica, Biosplice Therapeutics, BriOri Biotech, Centrexion Therapeutics, Eli Lilly, Dompe, Flexion Therapeutics Inc., Genascience, Gilead, GlaxoSmithKline, Kolon TissueGene, Novartis Pharma AG, Pfizer Inc., Regenosine, SFJ Pharmaceuticals Inc., Theralogix LLC, TrialSpark, WCG Analgesics Solutions, and Xalud Therapeutics. Member of Data Safety Monitoring Committees for clinical trials coordinated by ACI Clinical, Covance Inc., Cytel, ICON plc, IQVIA, and MiMedRx. Receive royalties from Elsevier (Editor, Rheumatology 7e and Editor-in-Chief, Seminars in Arthritis and Rheumatism) and Wolters Kluwer (UpToDate™). Stock options in BriOri Biotech, Regenosine, and Theralogix LLC. D. J. H.: Provides consulting advice on scientific advisory boards for Pfizer, Lilly, TLCBio, Novartis, Tissuegene, and Biobone. M.C.N.: No relevant relationships. W. W.: Employee and shareholder of Chondrometrics GmbH. C. K. K.: Consultant to Regeneron, LG Chem, Novartis, Xalud Therapeutics, and Express Scripts. He is the principal investigator for pharma-sponsored clinical trials to Abbvie, Cumberland, and GSK and DSMB to Kolon TissueGene and Avalor Therapeutics. X. S.: No relevant relationships., (Copyright © 2023 The Author(s). Published by Elsevier Ltd.. All rights reserved.)