Student Outcomes Survey: Self-Reported Graduate Model Review. Technical Paper

The National Student Outcomes Survey (SOS) collects information about students who completed their vocational education and training (VET) in the previous calendar year. The gathered information on the surveyed VET students includes their reasons for training, employment outcomes, satisfaction with training, and further study outcomes. The survey covers students who have completed a qualification (graduates) and those who have completed only part of a course and then left the VET system (subject completers). The National Centre for Vocational Education Research (NCVER) has conducted the survey with government-funded VET students annually since 1999. In 2016, the scope of the survey was expanded to report on the outcomes of graduates whose training was Commonwealth--or state-funded as well as fee-for-service graduates. These graduates were referred to as total VET graduates. An explanation of the difference between total VET and government-funded student outcomes can be found in appendix A. The expanded scope was applied to the 2017 survey for graduates (following a successful trial in 2016) and for the first time for subject completers and the series renamed VET student outcomes. At the time of sample selection, insufficient information is available from the National VET Provider Collection to identify 'actual' subject completers. Instead, a sample of potential subject completers is chosen, which includes students who are continuing in the VET system. The status of respondents is determined through the survey responses. As such, respondents to the SOS include a number of students who were sampled as subject completers based on administrative data reported to the National VET Provider Collection, but self-identify in the questionnaire as graduates. For many years these 'self-reported graduates' (SRGs) were categorised as graduates in survey outputs, because the self-report was deemed to be more reliable than the collection data. However, it became apparent that many SRGs were not, in fact, graduates. In response to this issue, in 2012, NCVER created a logistic model that predicted the eligibility of a SRG being an 'actual' graduate based on their personal and training characteristics (Braysher 2012). This model has since been run annually for each SOS to assign group membership to SRGs. Those SRGs that were not predicted by the model to be a graduate were re-assigned to their original subject completer status. One of the conditions of the initial analysis was that the model should be reviewed at least every four years to assess its ongoing validity and to make possible modifications should demographics and administrative data change and alter the predictive power of the model. The model was reviewed by NCVER in 2015 (unpublished). The review found some changes in data quality, but found no evidence that these changes were affecting the estimates. At the time no changes were recommended to the logistic model or graduate reclassification procedure, but it was recommended that the model be reviewed again at a later stage. In relation to data quality, the proportion of subject completers claiming to be self-reported graduates has declined significantly since 2007, particularly from 2015 to 2017, highlighting the improvement in the quality of the National VET Provider Collection data since the need to run the model arose (appendix B). Hence, it was appropriate to review the model again. This report presents the findings of an additional comprehensive review of the model and methodology.