In July the government published the specification for the first year of TEF subject pilots. Subject level pilots will be run in 2017/18 and 2018/119 before subject level TEF runs fully in 2019/20 (TEF Year 5). This also announced the use of a new teaching intensity measure.
On 7 September Jo Johnson announced a number of ‘refinements’ to the TEF methodology, arising from the TEF Lessons learned: summary report, and intended to make HE providers more accountable for teaching excellence. These changes come into effect for TEF Year 3 (ie immediately).
A summary of the arrangements for the subject pilots and the changes to the TEF methodology is outlined below.
TEF Subject Pilots
The design of the subject level TEF follows the design of TEF Year 2 (and incorporating the September announced changes), and covers all undergraduate provision. 30-40 providers will take part in the 2017/18 pilots, participating on one or both of two models; no ratings will be published.
The second level of the new Common Aggregation Hierarchy will be used to define subjects. This has 35 subjects. The same set of 10 TEF criteria that were used in TEF Year 2 will be used across these subjects
As at provider-level, each core metric will also be presented for a series of sub-groups (called splits) reflecting Widening Participation priorities.
Two models will be piloted:
- Model A: a ‘by exception’ model giving a provider-level rating and giving subjects the same rating as the provider where metrics performance is similar, with fuller assessments (and potentially different ratings) where metrics performance differ. Estimated that 39% of subjects across providers participating in the pilots will be assessed.
- Model B: a ‘bottom-up’ model fully assessing each subject to give subject-level ratings, feeding into the provider-level assessment and rating. Subjects are grouped for submissions, but ratings are still awarded at subject-level. Providers can choose to move at most one subject in and one subject out of each of these groups to help reflect the make-up of their structure but they do so at their own risk. See Table 4 in TEF subject pilot specification document for subject groupings. In this Model, provider-level ratings are based on provider-level metrics, provider-level submissions, and subject-level ratings.
In July the government also announced that a teaching intensity measure will also be piloted as a ‘supplementary metric’ in a small range of subjects in both models. This means it will be considered with the subject level submission to inform the holistic judgement. Two measures will be piloted for teacher intensity:
- A provider declaration of contact hours they provide, weighted by student-staff ratio (also taking into accounts provisions like placements, fieldwork and e-learning)
- A student survey on number of contact hours, self-directed study and consideration if the contact hours are sufficient to fulfil their learning needs.
The DfE plan to commission research to test aspects of subject-level TEF with a wide group of students. Professor Janice Kay, Provost and Senior Deputy Vice-Chancellor at the University of Exeter and Deputy Chair of the TEF Panel, will chair the main panel for the pilots. The same pool of panellists will be used for both models, but will operate differently.
TEF: Lessons Learned
On September 7th, the government released the TEF: Lessons learned summary report from the Department for Education (the full report to be published later in September), alongside a speech from Jo Johnson at UUK’s conference. This announced a number of ‘refinements’ to the TEF methodology which will take effect immediately and will also apply to the subject pilots. These include.
a) Weighting of National Student Survey (NSS) metrics
‘In order to achieve a more balanced assessment, the weight of each NSS metric will be halved for the purposes of determining the initial hypothesis’.
b) Part-time provision
In order to recognise excellence in part-time provision appropriately, providers with a majority of part-time provision will have the opportunity to submit an additional page of quantitative information alongside their metrics and the assessment procedure for providers with similar numbers of full-time and part-time students will be refined.
c) Very high and low absolute values
Whilst benchmarking will remain at the heart of TEF assessment, the flagging and benchmarking system has limitations at the extreme ends of the metrics. The top and bottom 10% of absolute values for each metric will be marked and taken into account in the calculation of the initial hypothesis, where a provider is not already flagged.
In addition to the above, Jo Johnson also announced that the government will strengthen the way that ‘TEF holds providers to account for delivering excellent teaching’. Measures include:
a) Longitudinal Education Outcomes (LEO) metric
Metrics derived from LEO data on graduate employment and salaries will be used as ‘supplementary metrics’, ie it will not affect the initial hypothesis but will be considered alongside the provider submission.
The two metrics are:
- The proportion of graduates in sustained employment or further study three years after graduation; and
- The proportion of graduates in sustained employment earning over the median salary[i] for 25-29 year olds1 or in further study.
b) Grade inflation metric
A new supplementary metric on grade inflation will be linked to the existing criterion on ‘Rigour and Stretch’ to aid assessors in making judgement in this area and ‘allow providers that are taking genuine steps to tackle grade inflation to be recognised for doing so’.
The supplementary grade inflation metric will record the proportion of firsts, 2:1s and other grades as a percentage of all classified degrees at that provider 1, 2, 3 and 10 years before the year of assessment. It will be collected by a mandatory declaration from all providers that apply for assessment.
c) Director for Fair Access to comment on whether ‘gaming’ has taken place
The Director for Fair Access will have the opportunity to determine whether ‘gaming’ has taken place (i.e. a significant alteration in a provider’s student profile since the last TEF assessment, that involves a reduction in the proportion of students from disadvantaged groups) so that panels can take this into account.
d) Power of referral
Where the assessment process suggests concerns about the underlying quality of a provider, the TEF Chair will have the right to refer the provider to the Office for Students with a recommendation that the OfS should consider an investigation to confirm whether the provider continues to meet baseline quality requirements.
Whilst these changes are presented as ‘refinements’ or ‘improvements’ in the Lessons Learned report, as several commentators have pointed out, they amount to significant changes, and move TEF further away from measuring ‘teaching excellence’ to a mechanism for implementing government policy and demonstrating ‘value for money’.
[i] The median salary for 25-29 year olds is currently £21,000. The figure is drawn from the ONS/HMRC publication Personal Incomes Statistics 2014-15, which is a survey of income data from HMRC and DWP and will be updated annually. This figure is below the starting salary for most modestly paying but socially valuable graduate jobs such as nursing, teaching or midwifery and the metric therefore records such outcomes as being equally valuable as higher paying professions such as banking or law. [TEF: Lessons Learned: Summary Policy document, Sept 2017, p 19]