“Lies, damned lies, and statistics”
Attributed to British politician Benjamin Disraeli, and popularized by American writer Mark Twain, this quote is often cited to illustrate how numbers can be used to persuade, make a case, repudiate or cast doubt on an argument. Within the context of the BAME attainment gap, and the use of performance metrics, the quote is apt.
Role of performance metrics
Higher Education has seen a growth in the use of, and reliance on, performance metrics to inform decision-making and strategy. Like many studies exploring discrepancies in the BAME attainment gap, RAFA began by examing existing data. In all our CPD sessions, across the three partner institutions, we presented academics with the attainment data for their students across the institutions, departments, schools and/or faculty level. We also drilled more deeply into the data by presenting academics with the attainment data for the programmes they taught on.
Credibility of data
In every CPD session, questions of the data’s robustness, reliability and credibility were raised as an explanation for the disparity. Academics also challenged the validity of the construct of race and ethnicity, preferring to foreground the debate by social class and socioeconomic status instead. Intersectional factors, such as attendance, students’ commute and family and caregiving responsibilities, were also raised as explanations for the disparity. Additionally, issues pertaining to the sample size, power size as well as reference to the types of statistical analysis conducted and what this might mean for the integrity of the data e.g. cumulative error, were also voiced.
We felt that if the findings of the project were to be taken seriously, we would need to address these points. To that end, we carried out an investigation into data at the University of Roehampton to address these concerns, which was undertaken in parallel with ongoing work with staff and students on addressing the barriers to the BAME attainment gap.
The approach adopted was to work with a team of data specialists to carry out an in-depth analysis of student attainment. The analysis looked at outcome variables, predictor variables, including demographic factors, prior attainment, engagement, module component and programme level factors.
The team worked on interrogating over 5000 data sets for variables believed to be valuable in addressing the BAME attainment gap. The data focused on 2018 graduates from the University of Roehampton, and investigated factors related to general attainment – year-on-year. Variables interrogated included anonymised ID, department, age, gender, socio-economic status (SES), where they lived (on/off site), distance, whether they were the first generation to attend university, home/EU/INT, disability, A levels or BTEC, tariff, student engagement, such as attendance, interaction with Moodle and the library), assessment and attainment .
The analysis involved the use of credible analysis tools, such as a generalized linear mixed model (GLMM), given the hierarchical nature of the data i.e. each student having results from multiple modules and even module components, and the effect of both student and module level predictors.
What remains is the issue of what information the data analyses’ allude to or occludes.
Below are links to resources which address some of the questions that arise concerning the power of data, such as who determines accuracy and what data is considered relevant and robust?
The OfS Data Strategy (2018-2021)
Jisc Using Analytics
Advance HE have used the following sources as a basis for some of the guidance on intersectionality:
Centred’s Intersectionality Literature Review
Hal Archives-Ouvertes France