HEAT-Key Stage 4 Track Impact Report: 2021 Exam Cohort

Introduction

Research has found that prior attainment, in particular at Key Stage 4 (KS4), is one of the key predictors of a student’s likelihood to progress to higher education (HE) explaining nearly all variation in HE progression (e.g., Chowdry et al., 2013). Disadvantaged students in particular are affected by this link between prior attainment and progression to HE, facing barriers to achieving the same educational standards as their non-disadvantaged peers, resulting in a risk to equality of opportunity (e.g., Teach First, 2017, Office for Students (OfS), 2018).

Owing to these findings, since 2019/20 the OfS requires HE providers (HEP) to evidence in their Access and Participation Plans how they intend to raise pre-16 attainment in schools (OfS, 2018). More recently, in February 2022, John Blake, the OfS’ Director of Fair Access and Participation, set out his expectations following his appointment and emphasized that outreach delivered by HEPs must focus on improving pre-16 attainment (OfS, 2022c). This has also been further backed up by a 2022 Insight brief on schools, attainment, and the role of HE (OfS, 2022a), and it is reflected in the OfS’ (2023) regulatory notice 1.

Pre-16 attainment-raising has also become a priority for Uni Connect (UC), with partnerships developing their approaches in 2022/23 ready for delivery from 2023/24 (OfS, 2022b). At the time of delivery to the cohort analysed in this report, the focus of UC was not yet on attainment-raising, but rather on targeted and strategic outreach¹. The cohort included in this report were due to sit KS4 exams in Summer 2021, which means their KS4 outcomes cannot be reflective of outreach delivered under the new attainment-raising focus introduced in 2022.

Both HEP and UC partnerships (UCP) have been recording a significant amount of attainment-raising outreach interventions on HEAT, identified by the use of the HEAT activity type² ‘Skills and Attainment’, used for recording activity designed to, directly or indirectly, raise outreach participants’ attainment. In the case of UC outreach, while a specific attainment-raising focus was not introduced until later, UC partnerships have nevertheless recorded significant engagement of pre-16 students under the ‘Skills and Attainment’ activity type. This suggests that activity delivered prior to the new attainment-raising priority being introduced can be expected to, at least indirectly, impact students’ attainment.

In this abridged report:

1. We first explore the composition of the cohort of students who participated in attainment-raising outreach delivered by a HEAT member provider before taking KS4 exams in Summer 2021.
2. Then we investigate the impact of participating in attainment-raising outreach delivered to the 2021 KS4 exam cohort on KS4 outcomes using a loose comparator group approach.

Footnotes and References are listed at the end of this page.

Methodology

On an annual basis, HEAT submits student records to the Department for Education (DfE) to be linked to records in the National Pupil Database (NPD). Through this linking exercise, KS4 exam attainment outcomes for these students are being made available in the Office for National Statistics’ (ONS) Secure Research Setting (SRS), including key attainment metrics and students’ free school meal (FSM) status. Once combined with activity and engagement data for these students from HEAT, this dataset can be used (1) to analyse which types of students participated in outreach and (2) to evaluate the impact of participating in outreach activities recorded on HEAT on students’ attainment at KS4.

To explore which types of students participated in pre-16 attainment-raising outreach, we use descriptive statistics with a focus on participation (see ‘Participant’ definition below) with a HEP or UCP, FSM status and prior attainment at Key Stage 2³ (KS2).

To evaluate the impact of engagement in attainment-raising outreach activities, we compare the GCSE (KS4) outcomes of a group of participants to a group of non-participants. In the context of this report:

• Participants are students engaged in at least five hours of ‘Skills and Attainment’ activities between Year 7 and Year 11
• Non-participants are students who engaged in only one single ‘General HE Information’ activity between Year 7 and Year 11⁴

KS4 exam outcomes are explored separately for students who took part in outreach with a higher education provider (HEP cohort) and for students who took part in outreach with a Uni Connect Partnership (UCP cohort) ⁵.

The GCSE metrics we use in this report are as follows:

• Percentage of pupils achieving 9-4 (standard pass)/9-5 (strong pass) in English and Maths
• Average Attainment 8 score

We explore these metrics for all students, students broken down by prior attainment band at KS2, and for students eligible for FSM.

Due to limitations in data access for this cohort, loosely formed comparison groups are used in this analysis⁶. This means controlling for factors known to impact KS4 attainment is only possible to do based on two available variables: KS2 prior attainment band and FSM eligibility. However, these two variables are critical in predicting performance at KS4.

Similar to KS4 attainment being an important predictor for HE progression, KS2 attainment is a key predictor for KS4 attainment; this is supported by national research (DfE, 2015), and by more recent analyses by HEAT and TASO (2021) and HEAT (2022). The breakdown by prior attainment used in this analysis therefore enables us to better understand how attainment-raising outreach affects pupils with different levels of prior attainment and removes some selection bias that may result from targeting.

FSM eligibility is also highly predictive of KS4 attainment, with national research reporting it to predict 23% of within school variation in GCSE attainment (DfE, 2015).

However, as prior attainment and FSM correlate highly, there remains a high degree of variation in the cohort, that cannot be explained by controlling for KS2 prior attainment and FSM eligibility together. These further influencing factors may include participants’ motivation, engagement in their education, and parental influence. This also poses a problem in relation to selection bias as students who are more motivated, or more inclined to improve their attainment, may also be more willing to participate in attainment-raising outreach.

Return to Introduction

Findings

Cohort analysis

The cohort in this report is based on outreach participants recorded on HEAT who were expected to have sat GCSE exams in Summer 2021⁷ and could be successfully tracked into the NPD. Over 90% of learners sent for tracking were found in the NPD’s KS4 cohort for 2021.

A breakdown of the participant and non-participant groups for the HEP and the UCP cohorts can be found in Table 1. The proportion of pupils eligible for FSM within the last six years is also provided; across all groups, approximately a quarter met this common criterion for disadvantage. This is in line with the national proportion of disadvantaged⁸ students in 2020/21 of 26.4%.

Table 1: Higher education provider and Uni Connect partnership cohorts by Participant and Non-Participant grouping

The two cohorts can be broken down into their level of prior attainment at KS2.

Key Stage 4 outcome analysis

The key finding of the KS4 outcome analysis was that larger proportions of FSM learners with ‘average’ prior attainment, who participated in attainment-raising outreach, achieved standard and strong passes in Maths and English than their non-participating peers. The identified attainment gaps between participants and non-participants were found for both learners who engaged in attainment-raising outreach at a HEP (Figure 2) and learners who engaged in attainment-raising outreach with a UCP (Figure 3). Both gaps were found to be statistically significant.

A more detailed outline of the key results across all attainment metrics explored can be found in the full report (members only).

Return to Introduction

Conclusion

A loose comparison group approach identified statistically significantly higher KS4 attainment (higher proportion of pupils achieving standard and strong passes in English and Maths), for FSM participants in the KS4 2021 exam cohort. In particular, statistically significant higher attainment was found for those with ‘average’ prior attainment at KS2, in both the HEP and the UCP cohorts, compared to FSM pupils who did not engage in attainment-raising outreach. This finding is particularly encouraging for outreach engagement of FSM pupils, a group at significant risk of equality of opportunity.

HEPs have been tasked to implement into Access and Participation Plans, and deliver, pre-16 attainment-outreach since 2019/20, with an increased focus since 2022. The results of our analysis show that, even though the cohort in this report had not yet had much exposure to attainment-raising activity under this new policy focus, students from disadvantaged backgrounds are benefiting from outreach provision designed to, directly and indirectly, raise attainment.

UCPs did not have a specific focus on raising students’ attainment at the time the students in this analysis were engaged in outreach. It is, nonetheless, promising to see a positive impact of UC delivery on KS4 attainment, even when the focus of delivery at the time was on targeted and strategic outreach. This suggests that activities designed to support students to make informed decisions about their future, may already be having a positive impact on their KS4 attainment.

The differences in KS4 attainment between participants and non-participants for both the HEP and the UCP cohort appear to have been driven by learners with ‘average’ attainment at KS2, which represented about half of the students engaged in outreach. Pupils with ‘low’ attainment at KS2 who participated in attainment-raising outreach between Year 7 and Year 11 did not experience the same attainment uplift observed for those of ‘average’ prior attainment. ‘Low’ prior attainers may have a much longer ‘distance to travel’, and they may be in need of attainment-raising outreach sooner than their peers with higher prior attainment. The support they require may also be different when compared to higher attaining students, suggesting that the attainment-raising outreach delivered to this cohort may have not been equally suited to the different prior attainment groups.

Analysis of the cohort participating in attainment-raising outreach also provided some insights in the targeting of attainment-raising activities. Across both HEP and UCP cohorts, the majority of all learners engaged in attainment-raising outreach had ‘average’ or ‘low’ prior attainment at KS2. Less than one-quarter had obtained ‘high’ prior attainment at KS2. This shows overall effective targeting of outreach by HEAT member providers to those students most at risk of failing to achieve at KS4.

In particular, we found that UCPs have, both in relative and in absolute terms, engaged more learners in ‘low’ and ‘average’ prior attainment bands. Arguably, these are the learners most in need of support with their attainment. This is not a new finding, and, as discussed in previous research by HEAT on targeting of attainment-raising outreach (HEAT, 2022), HEPs may engage lower proportions of learners with lower prior attainment because they are incentivised to recruit disadvantaged learners through their Access and Participation Plan ‘recruitment’ targets. UCPs may have greater freedom to carefully target those learners who are less likely to progress but are most in need of support with raising attainment at school.

Uni Connect is reaching the most educationally disadvantaged learners, which HEP may not reach on their own, and there is an evident concern that these learners will likely miss out without targeted initiatives such as Uni Connect. With the scaling down of Uni Connect funding, there is a danger that we may see opportunities for learners with lower prior attainment decline in the future.

The findings of this analysis support an association of taking part in attainment-raising outreach and improved KS4 attainment, in particular for FSM learners with ‘average’ prior attainment, which can be classed as Type 2, empirical evidence (OfS, 2019).

Return to Introduction

Next Steps

In order to progress this analysis to Type 3, causal evidence, the evaluation design will be reconsidered to feature closely matched comparison groups in future iterations of this report. Matching participant and non-participant groups on key variables known to impact attainment at KS4 will improve similarity between the participant and non-participant groups and contribute to more robust evidence on the impact of attainment-raising outreach above what otherwise might have occurred.

Future cohorts will include learners with more potential for exposure to attainment-raising activity under the new OfS priority of raising attainment of pre-16 learners for both HEPs and UCPs. This means the future HEAT-KS4 Track datasets will be drawing from periods when direct and indirect attainment-raising work was much more of a priority, providing a more robust base for evaluating the impact of that attainment-raising outreach.

As part of the new evaluation design, we will also explore ways of ‘disentangling’ engagement in outreach delivered by both UCP and HEPs and ensure ‘participant’ and ‘non-participant’ definitions are fit for purpose as cohorts who have benefitted from an increased focus on attainment-raising outreach are tracked into KS4 as part of the HEAT Track.

We look forward to future iterations of this report and to exploring the impact of an increased focus on outreach to raise attainment of those most at risk to equality of opportunity in KS4 attainment.

The next cohort, consisting of pupils who sat GCSEs in Summer 2022, will be available later this Spring, and will also see the return of the value-added Progress 8 scores, allowing us to take a closer look at pupils’ actual progress between KS2 and the end of KS4 compared to their expected progress.

In order to get the most benefit and value from HEAT, we encourage our members to use our range of tools and resources that support the design and execution of a robust evaluation design for their interventions.

Our tools support you at every stage, from planning interventions and evaluation, targeting schools and students, recording and monitoring progress, and evaluating outcomes. In particular, we encourage our members to use the ‘Evaluation Group’ field to attach comparator groups to activities and receive tracking data for non-participating students. This allows the membership to make use of local knowledge and insights to control for additional observed and unobserved variables that are not possible to account for in aggregate analyses.

For more information, please see the full report.

Read full report (members only)