Report

Pitfalls and possibilities in literacy research: A review of South African literacy studies, 2004–2018

Claire Biesman-Simons, Kerryn Dixon, Elizabeth Pretorius, Yvonne Reed

Received: 28 Mar. 2019; Accepted: 28 Oct. 2019; Published: 12 Mar. 2020

Copyright: © 2020. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background: Given the comprehensively documented literacy crisis in South Africa and the gaps in what is known about the effective teaching of reading and writing in schools, high-quality literacy research is a priority.

Objectives: This article evaluates South African research from two annotated bibliographies on reading in African languages at home language level (2004–2017) and South African research on teaching reading in English as a first additional language (2007–2018). It also aims to provide guidelines for addressing these weaknesses.

Methods: Examples of 70 quantitative and qualitative research studies from the annotated bibliographies were critically analysed, identifying key weaknesses in the research as a whole and examples of excellent quality.

Results: Weaknesses evident in the research reviewed, suggested greater consideration is needed to lay sound methodological foundations for quality literacy research. Three methodological issues underlying local literacy research that require greater attention are research design, selection and use of literature and research rigour. High-quality research examples are referenced but, for ethical reasons, examples of what we consider to be flawed research are described generally. Guidelines are offered for addressing these pitfalls that, in our view, contribute to research of limited quality. Since many universities require submission of a journal article as a requirement for postgraduate students, preparation for such an article is considered.

Conclusion: While this article is not intended to be a comprehensive guide, we hope it is useful to supervisors, postgraduate students and early career researchers currently undertaking, or planning to undertake, literacy research and to writing for publication.

Keywords: Literacy research; Annotated bibliographies; Review of research; Guidelines for research; Research rigour.

Introduction

As widely reported in both the popular media and in research reports (e.g. Howie et al. 2017; Isdale et al. 2017), South African primary school learners have performed poorly in international, regional and local literacy and mathematics assessments. While there have been numerous small-scale and large-scale research and development interventions at school level to address the problem and improve learner performance, less attention has been paid to initial teacher education at universities. A key finding from research conducted by the Council of Higher Education (2010) and the Initial Teacher Education Research Project (Deacon 2016) is that universities need to improve the quality and the curriculum focus of aspects of their initial teacher education programmes. As a response, in 2016 the Department of Higher Education and Training (DHET), with funding from the European Union, launched a 5-year Teaching and Learning Development Capacity Improvement Programme. One of its five projects is the Primary Teacher Education (PrimTEd) Project which aims to strengthen Foundation and Intermediate Phase mathematics and literacy courses in initial teacher education programmes.

This article arises from the work of the PrimTEd literacy group, of which the authors are members, which is tasked with strengthening the teaching of African languages as home languages and of English as a first additional language (EFAL) in teacher education programmes throughout the country. As part of its work, the authors reviewed South African research into reading in African languages at home-language level and in reading and writing in EFAL. The reviews have resulted in the compilation of two annotated bibliographies which are available on the open access PrimTEd website (https://www.jet.org.za/clearinghouse/primted/resources/language-and-literacy-resources).

In reviewing the literature in the field, Biesman-Simons and Dixon (the EFAL bibliography) and Pretorius (the African languages bibliography) identified several patterns that occurred across the reviews. While there were valuable sections in all of the articles, dissertations and theses reviewed, there was also recurring evidence of an incomplete understanding of what is required to conduct research that is sufficiently rigorous to be considered an addition to knowledge in the field. Given the literacy crisis in South Africa, the under-researched nature of literacy in the country’s schools, and the gaps in what is known about how to teach reading and writing, particularly in African languages, there is a great need for high-quality research.

The value of reviewing the literature that forms the basis of the annotated bibliographies is that there is a corpus of data that provides insights into not only what research is being done but also into evident strengths and limitations within this research. Rather than write a literature review identifying trends in local research regarding direction, focus and current findings, this article responds to one of the PrimTEd project’s aims of providing resources and support for those in higher education institutes involved in various aspects of literacy education and research, namely lecturers, postgraduate students, early career researchers and supervisors, to produce high-quality literacy research.

Our aims in this article are twofold. Firstly, we aim to describe key weaknesses in research design and the selection and use of literature and to unpack the concept of research rigour that we identified in the studies we reviewed for the annotated bibliographies. Secondly, we aim to provide suggested guidelines and actions for addressing these. By doing so, we hope this article will stimulate discussion and debate that will invigorate literacy research in South Africa. We acknowledge that all research is challenging and potentially messy and also that at universities, mentorship of novice research supervisors, novice researchers and early career scholars may be limited or non-existent, as a result of resource constraints.

The next section describes briefly how each annotated bibliography was compiled and then drawn on for this article.

The databases

As stated above, two annotated bibliography databases inform this article. The annotated bibliography on reading research in African languages comprises studies from 2004 to 2017. All relevant articles from local and international academic journals on the DHET’s list of accredited local and international academic journals were reviewed. Master’s dissertations and doctoral theses were also included. Most of the studies reviewed focus on reading research in any African language belonging to the family of southern Bantu languages (to which all nine official African languages in South Africa belong), although other African languages belonging to the south-western Bantu family (e.g. Herero) and eastern Bantu family (e.g. Swahili) were also included.

The EFAL annotated bibliography comprises research on reading and writing in EFAL in South Africa from 2007 to 2018. Articles were drawn from DHET-accredited South African journals. Included in the bibliography are citations of South African PhD and Master’s students’ research but at the time of writing, these had not yet been annotated. While most journal articles relate directly to reading and writing in EFAL, there are a few examples of annotations and citations that discuss teaching reading and writing in general but are pertinent to EFAL.

Table 1 provides a breakdown of the 70 texts that constituted the annotated bibliographies that we drew on for our analysis.

The evaluative framework

By its very nature, research is intended for public consumption, even if the public is comprised mainly of academics and stakeholders interested in the particular field in which the research is conducted. While research studies are published in academic journals, key findings sometimes appear in the media, and postgraduate dissertations and theses are available online and on library shelves in academic institutions where they can be accessed by other researchers in the field. Research should be held up to high standards. Poor quality research has a public face, and especially in the digital age, cannot be swept under the academic carpet.

During a PrimTEd meeting after the annotated bibliographies had been completed, the team members raised concerns about research-related issues that had emerged from a holistic reading of the studies. It appeared that these concerns were similar across both bibliographies and the team decided that a more intensive evaluation was required. During a 2-day workshop the four members of the team worked closely with the bibliographies identifying common pitfalls that emerged from the databases. Three of the authors are experienced supervisors, reviewers and researchers with strong publication records, and one of the authors is a novice researcher and doctoral student. One of the authors in the team comes from a linguistics and psycholinguistic background and has experience with quantitative methods, while three of the team members are steeped in critical and sociocultural approaches to language and literacy and have experience with qualitative research. The different disciplines, experiences and areas of expertise of the team members provided an equalising pool of theoretical and methodological perspectives to allow for fair evaluation of the strengths and limitations in the reviewed body of research.

Three key themes emerged from the review of the studies in the data set characterising methodological weaknesses that impact negatively on the quality of literacy research articles, dissertations or theses, namely: flawed research design, failure to select and use literature appropriately and lack of research rigour in both quantitative and qualitative studies. Examples of good quality research which illustrate what is possible for researchers to achieve were also identified. In the following sections we describe and discuss what we have termed pitfalls in research and offer some possibilities for avoiding them. The discussion is not intended to be exhaustive, given that each pitfall and how to avoid it could be the subject of a book chapter. Rather, it should be seen as complementary to the existing more detailed texts on research methodologies.

Flawed research design: Flawed writing about the research design

Having decided to embark on a research project, one of the initial major challenges for a researcher to address is that of research design. This is an important first step in the research process as a well-designed study provides the researcher with a logical and coherent frame within which to work. In the studies reviewed, the purpose of the research and the connections between research purpose and research questions were not always clearly stated. There were also instances in which the methods of data collection and data analysis selected for the study were not well described and occasions where the methods selected were not appropriate, or were inadequate, for addressing the research questions. Such weaknesses in either the conceptualisation of a project or in writing about a project (or both) raise concerns about the validity and reliability of the researcher’s findings (Cohen, Manion & Morrison 2011).

Extensive reading of literature in the researcher’s area of interest, together with literature on research design, is likely to assist with conceptualisation and coherent design of a research project. It may also be helpful to adopt a team approach to the conceptualisation of the research. Brainstorming and discussion with peers, colleagues and more experienced researchers or supervisors can help a researcher to develop a solid foundation for a study.

Inadequate or inappropriate selection and use of literature

Weaknesses in the selection and use of literature evident in a number of the articles, dissertations and theses reviewed for the annotated bibliographies also emerged as an area of concern.

Deciding which theories to use to inform one’s research and what literature to include in a literature review is a crucial part of research design and research writing but making these decisions can be quite a daunting task. As an indication of the challenges of working with literature, Kamler and Thomson (2006) title a chapter on this topic ‘Persuading an octopus into a glass’. Kamler and Thomson (2006) propose that the tasks of ‘literature work’ are to:

• sketch out the nature of the field or fields relevant to the inquiry, possibly indicating something of their historical development
• identify major debates and define contentious terms, in order to
• establish which studies, ideas and/or methods are most pertinent to the study and
• locate gaps in the field, in order to
• create the warrant for the study in question
• identify the contribution the study will make. (p. 28)

Additional functions of a literature review include providing a framework for relating new findings to previous findings, enhancing and acquiring the subject vocabulary and identifying recommendations for further research (Randolph 2009:2).

In the articles, dissertations and theses reviewed for the annotated bibliographies, four pitfalls that weakened the quality of research were identified in relation to the literature and theoretical frameworks that researchers selected and used. Each of these is described and discussed in this section.

Not reading enough in the field

Across both bibliographies there was evidence that researchers had not read enough in the field, resulting in the replication of research that has already been done or the failure to address – or at least show awareness of – key issues in the field. In many ways, the dissemination of research findings is a way to join a conversation in the field and to supplement an understanding of it. Not reading and drawing on published research undermines the integrity of the findings, and does not help to build, extend or change a field. Readers are referred to Botha’s (2018) thesis for an excellent example of an in-depth overview of local and international perspectives on reading and to Alcock et al. (2010) for a succinct synopsis, within the narrower constraints of a journal article, of phonological awareness research in general and of how research in an African context might add to and inform larger international debate.

While it is not possible to read everything, it is important to spend time searching for what has been published in the field, taking note of who researchers reference and being aware of how researchers align themselves with others in the field. It is also necessary to read original sources and research and to represent these accurately rather than relying only on second-hand sources, especially when critiquing a particular approach or findings. Developing a deep understanding of relevant literature is critical for new researchers who may not know the theoretical and ideological positions of many writers. The next pitfall is related to this first one.

Not reading the most appropriate research

This pitfall relates to work that did not sufficiently consult literature related to language and literacy research which may also have resulted in an imbalance between local and international literature. It was often the case that the literature reviews centred more on the work of educational theorists than of literacy researchers. While we are not claiming that the work of scholars such as Piaget, Vygotsky, Bourdieu or Bernstein has no value, or that their work does not provide rich frameworks for data analysis, we do contend that there are many other influential international and local language and literacy researchers whose work in disciplines as varied as cognition, neurolinguistics, linguistics, critical literacy, sociology and anthropology should inform literacy research (e.g. Freire 1972; Heath 1983; Janks 2010; McKinney 2017; Street 1984. See also Compton-Lilly, Lewis-Ellison & Rogers’ 2019 review of family literacy research). The problem with privileging educational theory is that it does not provide researchers with the lenses required for focusing on the processes of reading and writing, curriculum and pedagogical possibilities, classroom practices, assessment and testing and the debates in the field of literacy by reading and literacy researchers (e.g. Buckingham, Wheldall & Beaman-Wheldall 2013; Dehaene 2009; Freebody & Luke 1990; Green 2018; Oakhill, Cain & Elbro 2014; Seidenberg 2017; Snowling & Hulme 2005; Wolf 2007). There are also specialised sub-fields within literacy research which are important for literacy researchers to know about. Knowledge of literacy research in general, and of specialised sub-fields that relate to specific research projects, enables researchers to minimise the faulty generalisations that are evident in other research reviewed for the annotated bibliographies.

In addition to reading specialised literacy research rather than mainly general educational research, researchers need to think carefully about context. Although there is a paucity of literacy research in South Africa that focuses on primary schools (Pretorius & Spaull 2016) compared to the large body of international research, it is necessary to engage with quality local research (e.g. Makaure 2017; Matsatse 2017; Wilsenach 2016; Zimmerman 2014), noting its strengths and its limitations. Such critical engagement adds to the body of knowledge of literacy in this country, and enables researchers to work in particular areas, address specific research gaps systematically and build a more coherent field that feeds into teacher education and literacy research in more structured ways.

International research is valuable, but too often literature reviews treat it uncritically. Context matters and socio-economic, political, cultural and educational histories impact the kinds of language and literacy practices that are in place. While there are likely to be common findings across countries, it is important to consider how research from the political south speaks back to research from the political north (e.g. Makalela 2015; McKinney 2017; Pretorius & Mampuru 2007).

Not threading theory and literature through the research text

The sections entitled literature review in research articles and literature review chapters in dissertations and theses sometimes showed a lack of connection to other sections of the research. Dixon and Janks (2010) make the point that as a research project moves through its various phases towards completion, it becomes easier to work out where and how literature should be used:

Some of it provides secondary evidence for a claim; some of it may clarify an idea or can be referenced in passing; some of it provides direction for further research; some of it fundamentally reorganises one’s interpretation of the data; much of the later reading is confirmatory. It also becomes clearer where to fit it into the thesis. Not all of it has to go into the literature review. (p. 62)

They argue that researchers need to understand that the literature included in a completed thesis, dissertation or research article is:

… an account of the ideas that informed the project and that the project speaks back to … It is an articulation of how the work is positioned in the ongoing development of ideas in the field and how the researcher positions herself in the field. (Dixon & Janks 2010, p. 64)

Taking a ‘cookie cutter’ or ‘chunking’ approach to working with literature

Failure to thread the literature through an article, dissertation or thesis in a coherent way is likely to result in what we term a ‘cookie cutter’ or ‘chunking’ approach in which chunks are taken from a text and inserted in the researcher’s article or thesis as a direct quotation or as a paraphrased summary and then presented to the reader without further explanation or mediation. The researcher does not explain why these chunks are important to his or her analysis of data or developing argument, or how they are connected to each other.

While in some of the articles reviewed there was little or no evidence of any relationship among the texts reviewed by the researchers, the research design and the research findings, there were also articles in which pertinent summaries or quotations from literature were threaded through the presentation and analysis of data. See O’Carroll (2010) for a study in which the researcher is careful both to state explicitly the limitations of the study and to use literature in the discussion section to support the argument she is building.

Not threading the literature through the research writing may suggest a lack of understanding of the important role that specialist knowledge plays in research and the way that knowledge and the framing of knowledge contributes to an ongoing professional conversation within a particular domain of literacy and helps to inform prior, current and subsequent research. Mentors and supervisors of emerging researchers and reviewers of articles should pay more attention to the quality of literature reviews. There are many ways to write a literature review and to use literature in the various chapters of a thesis or sections of an article. What should be kept in mind are the questions why this literature, why this literature at this point in the text, and how do I want to use it?

Lack of research rigour

When preparing the annotated bibliographies, lack of rigour in much of the research reviewed emerged as a major concern. The longest section of this article thus focuses on the various ways in which this lack of rigour manifests in the data set.

Rigour in research refers to qualities of thoroughness, precision, accuracy and paying attention to detail. It also refers to adherence to norms and principles that underpin different approaches to research such as quantitative, qualitative or mixed method approaches. Achieving rigour is challenging for all researchers, but especially for inexperienced researchers and inexperienced supervisors of Master’s and PhD studies.

Failure to provide a detailed audit trail

An important aspect of rigour is providing for a detailed and transparent audit trail (Bassey 1999) which documents each stage of the research process (Gall, Gall & Borg 2007). An audit trail:

… endeavours to give access to others of the evidence which points to the trustworthiness of the study but also enables them to exercise their own creativity in finding alternative interpretations. (Bassey 1999, pp. 61–62)

An audit trail should include a well-thought-out rationale for the study and for the choices of theoretical framework and literature, explicitly stated details of the research site and participants, explanation of the data collection and analysis methods, clear analysis of the data which is informed by the theoretical framework and the literature reviewed, and clear presentation of conclusions (Bassey 1999).

In some of the studies reviewed, gaps in the audit trail included the following:

• The rationale for the selection of specific research instruments was not explicitly stated.
• In quantitative research, if several reading sub-skills were being assessed in reading tests (e.g. vocabulary, phonological awareness, word recognition, oral reading fluency), scant details about the sub-components were provided. For instance, the number of items included in each sub-component of the test was not indicated, or the text length, the number of words in a text or its grade difficulty level were not specified.
• When reading comprehension was assessed, there was often limited reporting on the length of the selected passages, the number of questions, the types of questions (e.g. literal, inferential and integrative) and the formats used (e.g. cloze, multiple-choice and full-sentence answers). For an example of detailed reporting, refer to Marx (2010) for a study in which reading comprehension tests designed by Grade 4 teachers were described in detail and then analysed in terms of appropriacy for learners at this grade level.
• In quantitative research, information about the reliability index of test instruments was not always reported. In ‘The patterns and prevalence of monosyllabic three-letter-word spelling errors made by South African First Additional Language learners’, Fleisch et al. (2017) provide a level of detail that should be made available to readers of research articles.
• Piloting test instruments before using them in the main study either did not seem to be a requirement or was not always documented in dissertations and theses. Piloting instruments (as well as data collection procedures) is part of research rigour. Like a dress rehearsal in a theatre production, having a trial run is necessary for picking up items that do not work or for identifying unanticipated problems. If instruments and procedures were piloted, it is particularly important that a section in the research report needs be dedicated to the pilot, and how or why the instruments and procedures worked or did not work and what changes were made.
• Inadequate contextual information was provided. Given the complex nature of schooling in South Africa, the diverse teacher and learner population and the socio-economic and structural inequalities inherited from apartheid, literacy research in the southern African context requires descriptions that situate the research within a clearly defined context and within which possible limitations as a result of this diversity can be clearly identified. This is particularly important in small-scale studies because of the key role that context plays in adding richness to the data, positioning findings and providing a framework for research that could be replicated in similar contexts. For an example of a richly contextualised study, see Prosper and Nomlomo (2016). Their explanation of the context of the study, the participants’ position within the study and how the context informed the purpose of the research provides a strong foundation for responding to their research questions.
• Insufficient information was provided in regard to procedures used for data collection. In literacy research, reporting when, how and by whom data was collected is important not only in terms of replicability but also in terms of how the outcomes are interpreted. Probyn’s (2016) description of the data collection process in her study of language use in Grade 8 science classrooms in the Eastern Cape is an excellent example of detailed, relevant and interesting reporting.

Two examples illustrate the importance of addressing when, how and by whom data is collected. Firstly, learner performance can be affected by when learners are assessed. Assessing Grade 1 learners’ vocabulary knowledge in Term 1 of the academic year ought to yield different results from a vocabulary assessment in Term 3. Secondly, what teachers do and when they do it is affected by the academic year and the demands of the curriculum. For example, the South African curriculum stipulates that shared reading in Foundation Phase should be done for 15 minutes, three times a week, with each session centred on a different way of using the story to develop language or literacy skills and knowledge (Department of Basic Education [DBE] 2011). A study that focuses on how teachers use shared reading will thus have to observe the teacher at least three times in a week, otherwise data collection procedures will not do justice to the topic being examined.

Underestimating the demands of qualitative research

An assumption, more frequently visible in the work of inexperienced researchers, is that qualitative research is easier to undertake than quantitative research because data from a qualitative study do not require statistical analysis. This is exacerbated by the case that very few academic departments in the human and social sciences at South African universities teach quantitative research methods at undergraduate (or even postgraduate) level (except perhaps in the fields of economics and psychology), so there is little support for students who opt to undertake quantitative research to develop the requisite statistical and methodological knowledge and skills. However, qualitative research is demanding in several ways. According to Cohen et al. (2011), a qualitative researcher needs very good verbal skills, the capacity to describe and interpret data and the ability to read between the lines. The researcher needs to explain why a qualitative approach was appropriate, why particular data collection methods were selected and what informed the approach to data analysis (Cohen et al. 2011). Any potential limitations and personal involvement must be articulated, as must what was done to address each of these. Identifying what constitutes rigorous qualitative research and how best to conduct such research is challenging, even for an experienced researcher. Two examples of high-quality qualitative literacy research are Van der Mescht’s (2013) PhD study and Mather and Land’s (2014) journal article on educators’ understanding of reading skills.

Across both bibliographies there were large numbers of small-scale studies, many of which were flawed in various ways. For example, in a 5-year period, 15 out of 23 of the entries in one of the bibliographies involved small-scale studies, and, of these, more than half were problematic with regard to one or more aspects of research rigour.

Many of these small-scale studies took the form of case studies. When well designed and executed, case studies are valuable for the richness of the data they can provide, but many of the studies reviewed were found to be problematic. Areas of concern identified by the reviewers include the use of a single research instrument (i.e. absence of triangulation), lack of a rationale for the choice of case study, insufficient explanation of the context within which the research was undertaken and of the reasons for situating the research in such a context, and researchers making large claims and unwarranted generalisations despite the small size of the study. Due to the small size of case studies, rigorous research is essential in order to produce valid findings and interpretations that can lead to what Bassey (1999) terms ‘fuzzy generalisations’. Guzula, McKinney and Tyler’s (2016) study is an example of rigorous case study research.

Limited understanding of ‘size’ and its relationship to rigour

The quality of a study is not determined by its size, the number of participants or number of research sites. Rather, it depends on how rigorously the research is undertaken within its specified parameters. Similarly, the size of the study should be guided by the requirements for how best to respond to the research questions. For instance, if the research question relates to Grade 4-literacy practices in South African classrooms, a case study involving a single teacher is unlikely to be able to address such a question fully.

We organised the data set into four categories of study size depending on the number of participants involved and the number of schools that served as research sites. Participants involved were either learners, teachers or other officials in education, such as principals, district officials or subject advisors. The four categories of study size were:

• Small: research involving 100 or fewer participants or one or two schools.
• Medium: research involving between 100 and 499 participants or 3–10 schools.
• Large: research involving 500–1000 participants or more than 10 schools (these can involve up to 60 schools, with limited numbers of participants from each school).
• Very large: research involving over 1000 participants or more than 20 schools (usually well over 200 schools).

Size is obviously a relative concept, so these categories serve as rough guides only. The large and very large literacy studies in South Africa have typically derived from team-based, well-funded international or national projects where researchers have had access to large data sets and have worked individually or jointly with team peers. The very large studies come from the Progress in Reading Literacy Studies (PIRLS) involving 12 000 and more learners¹, while the first large-scale randomised control studies undertaken by the DBE in North West (Setswana home language) and in Mpumalanga (EFAL) have both involved the assessment of over 4000 learners in each successive iteration (Taylor et al. 2017).

Generally for Master’s and doctoral studies students are expected to produce individual research, and the articles that derive from postgraduate research are written individually or co-authored with a supervisor. Because large and very large studies are team-based and funded, there tend to be methodology experts in the project team with expertise in quantitative or qualitative approaches who aim to ensure that the studies are well designed and rigorously conducted. The same checks and balances are not always built into small-scale individual studies. While the importance of small-scale studies must not be underestimated, every effort should be made to ensure that they are carefully designed and meticulously conducted.

Lack of data triangulation

For research to be rigorous, it must be conducted in a manner that produces useful and reliable data. The need for more rigorous triangulation of data was also identified as an area of weakness. Lack of triangulation was particularly concerning in qualitative research involving classroom or teacher-based studies, where data were collected from a single source (e.g. teacher interviews) or a single instance (e.g. one lesson observation) and conclusions were then drawn about how literacy was enacted in the classroom, without being corroborated by data from other sources.

Claims about the efficacy of interventions (e.g. teacher workshops, literacy circles, after-school classes) were often made solely on the basis of self-report questionnaires or interviews without conducting classroom observations or capturing learners’ literacy performance before and after the intervention, to verify that reported changes were occurring as a result of the intervention.

Lack of data triangulation can lead to opinion-based rather than evidence-based research, or to making claims not substantiated by the reported data. For instance, if teachers are self-reporting on their classroom practices, this information does not necessarily offer a reliable account of their practices as it is opinion-based. For the research to be considered rigorous, teachers’ self-report data would need to be complemented by classroom observations to ascertain whether there was a similarity between teachers’ espoused and enacted practices. Kruizinga and Nathanson (2010) offer an example of comprehensive triangulation of data.

Limited analysis and interpretation of data

For research to be accepted as rigorous, it is important that data are carefully and appropriately analysed. In some of the quantitative studies reviewed, there was evidence of limited understanding of the requirements of quantitative analysis (e.g. claiming differences when there were no statistically significant differences or claiming relationships and interactions between variables without employing inferential statistics), insufficient background knowledge as to how to approach quantitative analysis and making grand claims from a small data set. Consequently, there was a lack of meaningful discussion and analysis in response to the context, research questions or to the literature and theory that framed the research. On the other hand, failure to take into account the researcher’s own position in some of the qualitative studies reviewed limits the rigour of the analysis and interpretation.

Failure to understand the role of researcher subjectivity

All researchers need to consider how the researcher’s subjectivity can influence both the research design and the interpretation of findings. This is particularly the case in qualitative research where a researcher’s interpretations are not guided by predefined rules, as is often the case for quantitative research, and where personal interaction with participants may be required (Davies & Dodd 2002). Lack of rigour may be the result of approaching research with preconceived assumptions and anticipated findings, having a personal relationship with research participants or selecting a specific research design due to personal preferences rather than establishing the most appropriate approach (Gioia, Corley & Hamilton 2012). While subjectivity cannot be avoided, it is key that checks and balances are in place. A critical reading of a first draft of a research proposal, thesis chapters or a journal article by more experienced researchers or supervisors is likely to be helpful. Potential limitations of the study design, or of the interpretation of findings arising from the researcher’s subjectivity, should be acknowledged in the final text.

From the genre of the dissertation or thesis to the genre of the research article

Increasingly, submission of a journal article for publication is a graduation requirement at South African universities and, if not mandatory, at least strongly encouraged. Thus, regardless of whether postgraduate students will be or would like to be academics for whom publishing is an essential part of working life, most will need to write a journal article informed by their examined research project. While a few of the characteristic features of a thesis and a journal article are similar, for example both focus on the author’s research and aim to add to current research conversations or to start new ones, there are more differences than similarities, beginning with length and anticipated readership.

For a dissertation or thesis, the initial readers are the supervisors and examiners, and the text they read will vary in length from approximately 20 000–25 000 words (at Master’s research report level) to approximately 80 000–100 000 words (at PhD level), with differences in length across faculties within a university and across universities. A journal article is likely to attract a wider informed readership and will be much narrower in focus and more succinctly written than a thesis. The length of articles accepted by journals varies, but for those in which literacy research is likely to be published, a word limit of between 6000 and 8000 is common. Readership and length have implications for what a researcher chooses to exclude and include, and to background and foreground in the article. While binaries can oversimplify, Table 2 summarises important contrasts between the expectations of thesis examiners and journal article readers.

Manuscripts submitted to journals are likely to be rejected if they are too ‘thesis-like’ or if they are not well suited to the conversations that take place in particular journals. An essential question for authors to answer before submitting a manuscript to a journal is: Who is the imagined reader? (e.g. a reader interested in translanguaging, a reader interested in critical literacy or a reader interested in the transition from learning to read to reading to learn). A second key question is: What is the contribution I wish to make to the conversation on this topic in this journal? Thomson and Kamler (2013:164) suggest that a researcher makes a publication plan which responds to questions such as which parts of the research he or she wishes to prepare for a journal article and for which journal he or she is preparing the article. To begin to answer these questions, researchers need to be thoroughly familiar with the content of what has been published in a particular journal in the preceding 3–5 years and with the house style of the journal. As it is not always easy for less experienced researchers to distinguish accredited journals from predatory ones, we suggest that journal choices be discussed with experienced colleagues or the university’s research office, and the list of accredited articles, recognised and regularly updated by the DHET, be consulted. (Although published 8 years ago, Uysal 2012 also provides a useful summary of international language and literacy journals that is still relevant.)

Conclusion

The two annotated bibliographies that we have drawn on for this article include a number of examples of quality research, in addition to those cited in this article. However, we chose to focus on the limitations in the local research on reading in African languages and EFAL, and to offer guidance for addressing these, as a contribution to enhancing the quality of research on aspects of teaching and learning literacies. Producing quality research enables literacy researchers in South Africa to assist in addressing the literacy crisis in education locally, but it also enables researchers to publish in high impact international journals and, in doing so, to contribute to global conversations in which the political south talks back to the political north. It is hoped that the issues raised in this article will promote constructive debate and reflection on ways to advance and improve literacy research on the continent.

Acknowledgements

This publication has been developed through the Teaching and Learning Development Capacity Improvement Programme which is being implemented through a partnership between the Department of Higher Education and Training and the European Union. The contents of this publication are the sole responsibility of the authors and can in no way be taken to reflect the views of the European Union.

We declare that we have no financial or personal relationships that may have inappropriately influenced us in writing this paper.

This is a jointly conceptualised and written article.

K.D. is responsible for the idea and structure of the article and this was then planned collaboratively. Y.R. wrote the opening sections, K.D. the section on literature reviews and co-authored the conclusion with Y.R., C.B.-S. and E.P. wrote the sections on rigour. All authors gave feedback to each other over drafts, proofreading and edited before submission.

This article followed all ethical standards for research without direct contact with human or animal subjects.

This publication has been produced with the assistance of the European Union and the Department of Higher Education as part of the Primary Teacher Education Project.

Data sharing is not applicable to this article as no new data were created or analysed in this study.

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.

References