Menu
Close
- Research Blog
- Nottingham Trent University: Case study evidences how a quasi-experimental design is a useful tool for outreach evaluation
Our Member Spotlight showcases NTU’s case study and how they utilised HEAT tracking data to assess the impact of pre-16 outreach interventions on Key Stage 4 (KS4) GCSE exam attainment, using a quasi-experimental design (QED). Members can access NTU’s full case study on File Store
A QED is a research method that can be used when randomisation is not possible (which is often the case in outreach evaluation).
Instead of research participants being randomly assigned to a group (treatment group being exposed to the intervention and control group not taking part in the intervention), naturally occurring groups or pre-existing conditions are used to form treatment and control groups.
Creating a Quasi-Experimental Design
NTU wanted to understand if their outreach activities have had any impact on raising attainment in the participants taking part in them, and the aim is to show a practical example of the process which other members may wish to replicate with their own data. Randomised Control Trials (RCTs) are considered the most robust form of evaluation which are the most likely to produce Type 3 (causal) evidence according to the Office for Students (OfS) standards of evidence and evaluation tool.
As is commonplace throughout the widening participation sector, NTU did not have sufficient non-participant data which can easily be used to create control groups. Randomisation is also problematic when delivering interventions in schools for several reasons. These include obtaining consent, getting schools to ‘buy-in’ to the methodology, logistical constraints around splitting classes/year groups, as well as ethical considerations.
Understanding Case-Control Matching
Despite the lack of random assignment, well designed QEDs can still allow causal inferences about the relationship between the independent and dependent variables to be made, albeit with much less certainty compared to where there is randomisation. For a small number of current attainment-raising activities, NTU found there had been some progress in conducting RCTs. However, this has been on a small scale with individual schemes, and whilst a positive development, it has challenges, is resource intensive, and could not be replicated for all outreach activities.
To try and evidence the overarching impact of outreach activities on attainment raising, NTU used participant data collected and stored on the HEAT database as part of a QED (specifically case-control matching) to look for any differences in attainment for our participants. Case-control matching is a QED which allows us to control for other factors that influence pupil outcomes – known as confounding variables. A participant would be matched with a non-participant who has similar characteristics. The matching aims to ensure that any differences observed (in outcomes) between the two groups are more likely to be due to being a participant rather than the other variables, by minimising the effects of the confounding variables. Case-control matching is regularly used in observational studies to reduce (although not necessarily eliminate) selection bias.
Experience and Findings
NTU’s case study explores their experience on coding ‘Descriptors’ and ‘Programme Title’ fields using the HEAT Activity Typology, and using the ‘Evaluation Group’ field to create groups of pupils to be compared and coded in respective groups (belonging to treatment and assigned control groups) that were later analysed. The resulting report guides members through this process, from identifying potential treatment and non-treatment groups, preparing the participant data, coding data on the HEAT system, and explains how they undertook this work in a practical way that may be of benefit to others.
Results
On GCSE (KS4) attainment, NTU found that 76% of pupils who participated in an outreach activity during their secondary education achieved 9 to 4 (standard pass) in their GCSE Maths and English exams compared to 65% of those who did not take part in outreach during this period of their education. Likewise, 56% of pupils who participated in an outreach activity during their secondary education achieved 9 to 5 (strong pass) in their GCSE Maths and English exams compared to 45% of those who did not.
Although NTU could not prove a casual link between their interventions and KS4 achievement, using a QED methodology they were able to show that their outreach interventions are associated with beneficial results, which meet Type 2 (empirical enquiry) OfS Standard of Evidence.
Their study provides an excellent example of whilst QEDs can appear quite daunting, they can be used to provide quality evaluation of outreach interventions using HEAT data. We hope this can encourage other HEAT members to generate and analyse their own Key Stage 4 Attainment track data in a similar way.
With HEAT’s support and the resources they have available, it has been a straightforward process to develop and one which we plan to incorporate into our evaluations going forward. We hope that this case study will encourage other members to view and analyse their own data in a similar way. Indeed, larger-scale national analyses using this approach could provide further evidence of the impact of outreach activities on pupil attainment, and be of interest to the sector.
Nottingham Trent University, Key Stage 4 Attainment Impact Analysis Case Study
HEAT Resources
NTU found undertaking this study with the support of HEAT and our resources a very positive experience, and referred to three main resources to support their evaluation work on this study, which are available to members on File Store:
- Comparator’s Working Group Resources: All notes and slides from the Comparators Working Group 2023/24 working group webinars
- Case-Control Matching in SPSS: A step-by-step guide on how to perform case-control matching in SPSS for the purpose of creating matched comparison groups for the evaluation of outreach activity
- HEAT Activity Typology: A framework for members to follow when creating Activity records to ensure consistency in the aggregate dataset. Provides guidance on all fields including Category, Exchanges, Frequency, Type and Location
Thank you to Nottingham Trent University for allowing us to showcase their findings. Please reach out to researchandinsights@ntu.ac.uk for any questions on their evaluation work.
Back