AI and Psychometrics: Epistemology, Process, and Politics

If psychometrics has long concerned itself with validity, reliability, and fairness, then what could psychometrics learn from the cybernetic theories of AI? Through engagement with Burstein's (2023) Responsible AI Standards, this paper unpacks some paradigmatic differences between psychometrics and cybernetics, points to how recursivity and contingency are both a challenge and opportunity for psychometrics, and how this matters epistemologically, ethically and politically. Following these epistemological differences, the paper raises ethico-political concerns with the promise of the "human-in-the-loop".