Financial education has become the popular antidote to the widespread financial literacy "problem" in the United States. Certain demographic groups (i.e., young adults, the elderly, women, and people of color) have been consistently identified as "financially illiterate." However, substantial increases in financial education – many targeting these specific groups - have achieved little, as research consistently finds that improvement to financial literacy in the U.S. remains elusive. The current study examined the validity and fairness of financial literacy assessments: specifically, whether test score interpretations and uses may be unfair due to measurement bias. Validity and reliability measures were analyzed, and Differential Item Functioning (DIF) analysis was conducted to identify potential measurement bias for the six financial literacy questions included in the 2018 National Financial Capability Study. Item responses were examined by participant age, race, and gender. The study found a lack of validity evidence regarding internal item consistency and item correlation, suggesting that the items measure different or multiple constructs. Furthermore, Differential Item Functioning (DIF) was indicated for three of the six items: Items two, four, and five. Large DIF was indicated for item two for the 65 and older focal group (β = -.13). Moderate DIF was found for item two in favor of the 18-24 age group (β = .06). Item four indicated large DIF for the 18-24 age group (β = -.10) and in favor of the 65+ focal group (β = .11). Items two and five indicated DIF for the Black/African American focal group (β = -.08 and -.06, respectively). Large DIF was found for item five in favor of the 18-24 age group (β = 0.09) and moderate DIF was found in favor of females in the gender group (β = 0.07). Test developers and users may find these items unsuitable for their target population(s).