Alma Economics was commissioned by the Department for Education to review the literature on interventions to improve numeracy skills among adults with below Level 2 qualifications. For this purpose, we developed a map summarising current evidence based on the systematic evidence review we conducted.
The evidence map categorises the research studies into four broad themes relating to focus of the study and a range of outcomes for individuals. Each cell includes the number of papers that correspond to a specific theme and a specific outcome for individuals. By clicking on a cell, a tab appears. This tab includes the titles of the research studies exploring the relevant research theme (x-axis) and the type of outcome that was the focus of the study (y-axis).
Supplementary information includes: (i) the abstracts or brief executive summaries of the reviewed sources that included such sections, (ii) the author(s) of every piece of evidence, (iii) publication date, (vi) a web link to the study of interest. Our map is interactive; it includes filters that allow users to swiftly explore the evidence of interest. Users can choose between the following filters: type of publication, date of publication, the primary methodology used and an overall quality assessment of the paper's relevance, credibility, and methodological quality.
Note: One study might examine more than one primary research theme or outcome meaning that the same study might appear in more than one cell.
Disclaimer: Several sources were retrieved from academic journals or other databases that do not provide their content for free. As a result, readers may not be able to access these sources directly from this evidence map. If interested, the readers could either examine their subscription options or contact the providers of the sources or relevant libraries to explore the possibility of accessing them.
The quality assessment of the evidence for this review was based on three main categories: (i) credibility, (ii) methodology, and (iii) relevance of the study. This is a judgement not on the quality of any single piece of research, but of the quality and relevance of the evidence provided for answering the research questions of this review.
For each category, we assigned a score from 1-3 (where 1 is the lowest score and 3 is the highest). This scoring guides the synthesis of the evidence produced in the full report, where we emphasise the highest quality evidence. The methodology score draws on the Maryland Scientific Methods Scale, but has been adapted to allow us to effectively categorise the large number of sources in the literature applying qualitative approaches and mixed-methods.
Our suggestion is to interpret with great caution any piece evidence acquired from sources with quality score 4 or below. That, because a source with an overall score of 4 or 3 will have a score of 1 in at least two of the three main categories, whereas it cannot have a score of 3 for any of the three categories. Additionally, the sources with an overall quality score of 5 should be reviewed with some caution because they could have a score of 1 in up to two categories.
Below there is a table describing in detail the appraisal criteria for each of the main categories.
| Category | Description | Score |
|---|---|---|
| Credibility | Is the study coherent? Can findings be trusted? Does the author consider study limitations or alternative interpretations of the analysis? Has the study been peer-reviewed? 1 = Study has not been peer-reviewed, with conclusions drawn from limited data or theoretical discussion. Lack of transparency around data and no discussion of data quality. No discussion around assumptions made. 2 = Study is unpublished, or study is informally published as a working paper/research report by a reliable source. Limited discussion around sources, information and data quality, or alternative interpretations of research findings. Study focuses on an ongoing intervention with adequate discussion around assumptions made. 3= Study is published in a peer-reviewed academic journal. Study discusses information quality, sampling decisions and other aspects of the methodology. Study focuses on a completed initiative. | 1-3 |
| Methodology | How robust is the evidence to contribute to our review? 1 = Methodology is not fit-for-purpose and relies on before-and-after or cross-sectional comparisons with no use of control variables. (very similar to Maryland Level 1). This score will also be given to qualitative studies with unclear/inadequate sampling strategies. No discussion of why the chosen design and method are well-suited to answering the research question. 2 = Methodology is fit-for-purpose and relies on adequate control variables, though important unobserved differences may be remaining (Maryland Level 2). This score will also be given to high-quality qualitative studies (surveys, focus groups, case studies) with robust sampling strategies. Some discussion of why the chosen design and method are well-suited to answering the research question. 3 = Methodology exploits quasi-randomness in treatment or explicit randomisation into treatment and control groups [1]. The study provides clear evidence on comparability of treatment and control groups. Extensive discussion of why the chosen design and method are well-suited to answering the research question. Literature reviews and meta-analyses will also receive this score. | 1-3 |
| Relevance | Does the study help to answer the research question? 1 = The research question or hypothesis is not directly related to the proposed research questions or considers interventions in non-OECD countries. 2 = The research question or hypothesis is only somewhat related to the proposed research questions or considers interventions in OECD countries. 3 = Study addresses an intervention in the UK and/or the research question or hypothesis is directly relevant to the proposed research questions. | 1-3 |
| Overall judgment | Considering the above categories, what is the overall judgment? | 3-9 |
Engagement: research relevant to informing (e.g., media campaigns and communication channels) and incentivising (e.g., rewards or other benefits) adult learners to participate in and complete numeracy courses. Also, research questions relevant to the participants' motivations to learn and persist were included in this theme. Delivery: research relevant to the effectiveness of different channels to deliver adult numeracy learning provisions including workplace learning, digital provision and e-learning, and community- or classroom-based delivery approaches. Teaching: research relevant to the effectiveness of various teaching practices including the use of digital technologies, home-learning practices, and progress monitoring techniques. Additional support: research relevant to the need and effectiveness of additional support measures. Such measures included provisions for adult learners with SEND or other learning difficulties, and provisions for individuals with maths anxiety. Additionally, research questions examining the effects of peer support, family support (e.g., support from the learners' family environment as well as support for learners with care responsibilities), and financial support.
We have categorised sources into six different publication types: book chapters, conference papers, journal articles, reports, and any other type of publication.
Book chapter: sources that were a chapter (or more) of a published academic book.
Conference paper: research papers written as parts (or outcomes) of conference lectures
and have not been published in an academic journal.
Journal article: research papers published in academic journals.
Report: programme evaluations and research conducted for/by governmental bodies,
foundations, and other non-governmental organisations.
Working paper: research papers that have been completed, but not officially published
in an academic journal.
Other: any other type of publication that does not fit the previously mentioned
types.
We also categorised the outcomes that were studied in individual papers into broad categories. Each piece of research could address multiple themes.
Engagement: the degree of involvement, persistence, and motivation exhibited by adult
numeracy learners or potential learners.
Completion: the extent to which adult learners successfully completed numeracy courses
and the retention/drop-out rates.
Attainment: the performance of adult learners in summative assessments compared with
their performance in diagnostic assessments or achieved formal qualifications after participating in
learning activities.
Personal: other aspects of the learners' personal and everyday life that was affected
by adult numeracy courses. This could include interpersonal relationships (e.g., participating in
conversations or helping their children with homework), the use of numeracy skills for transactions and
other needs, and changes in self-confidence and feelings of self-efficacy.
Employment: outcomes relating to learners' professional life including new employment
opportunities, promotions, salary increases etc.
