AAAL 2025: Invited Colloquium

Researcher reflexivity and positionality in quantitative and experimental research

Conveners:

Abstract:

Researchers in the social sciences grapple with a fundamental tension: their pursuit of knowledge about the world versus their embeddedness within it. This tension manifests differently in quantitative and qualitative paradigms: quantitative researchers aim for objectivity and generalizability by minimizing their (the researchers’) role, while qualitative researchers embrace their influence on data production and interpretation through the practice of reflexivity. 
Reflexivity is the act of examining how one’s own values, beliefs, and past experiences may shape one’s research process (e.g., Consoli & Ganassin, 2023; King, 2024; Starfield, 2013). Although reflexivity has sometimes been argued to be at odds with the notion of research objectivity advanced in quantitative approaches (Harris, 2021), some quantitative researchers are beginning to introduce elements of reflexivity in their work (Godfroid & Andringa, 2023; Kutlu & Hayes Harb, 2023; Perette, 2023). They position themselves more clearly in their research, sometimes by writing positionality statements, such as those included in the 2023 special issue of Applied Psycholinguistics, Towards a Just and Equitable Applied Psycholinguistics (Kutlu & Hayes Harb, 2023). 
Practicing reflexivity is relatively new in quantitative approaches in applied linguistics (see Jamieson et al., 2023, for a similar development in social and personality psychology). In our view, it represents an interesting development alongside the increasing emphasis on transparency, reproducibility, and generalizability associated with the open science movement. Indeed, it raises the question of how the subjective and objective can co-exist in the same project.

This colloquium explores the degree to which quantitative and experimental researchers can and should address questions of identity and positionality in their work. It explores what the practice of reflexivity in quantitative research could look like and examines its compatibility with ideals of transparency and objectivity. The speakers in this colloquium are all trained primarily in quantitative and mixed-methods research, but through their life and/or professional trajectories, have been invited to reflect on the question of reflexivity in their scholarship.

Sampling and generalisation practices in second language acquisition research

Sible Andringa, University of Amsterdam
Anne-Mieke Thieme, University of Amsterdam

One important decision that researchers make in study design concerns the choice of sample and the specific research context in which a study is conducted. This step may heavily reflect a researcher’s background and interests, as well as strongly determine the outcomes of a study. In a perfect positivist research world, researchers strive for statistical representativeness, a match between a study’s sample composition and the population that is targeted given the study’s goals or research questions (Henrich et al., 2010; Kruskal & Mosteller, 1979b). Together with Aline Godfroid, I’ve questioned current sampling practices (Andringa & Godfroid 2020; Godfroid & Andringa 2023). At the same time, I’ve struggled with tensions between practicality and feasibility on the one hand, and study quality and generalizability on the other hand in my own work, and regularly accepted sampling decisions that are far from perfect. Many of us probably have similar experiences. The goal of the present study was to understand how we deal with this as a field, to initiate discussions on how we should deal with this.

We selected all articles from eight high impact, general scope SLA journals published within one year, and systematically analysed the relationships between 1) targeted populations given a study’s research questions/goals, 2) the recruited participant samples and 3) the presence of reflection on sampling choices. In doing so, we aimed to uncover researchers’ awareness of the impact that sampling choices exert on study outcomes. We found that reflection on sampling and context was quite absent, in both quantitative studies and qualitative studies. In this presentation, we will explore why such reflections are often absent and how reflexivity and positionality statements might help us to improve interpretation of study outcomes. Ironically, then, this study constituted an attempt to objectively and quantitively study subjectivity and reflexivity in SLA research.

Beyond deficit views: The importance of reflexivity in language science research

Ethan Kutlu, University of Iowa

Language science has historically disregarded variability, including differences among individuals and variations in input (Kutlu & Hayes-Harb, 2023). Early research on second language acquisition followed this trend, largely due to SLA theories being based on comparisons with monolinguals. Moreover, methodologies that are used to test these theoretical approaches heavily depend on discarding variability. Participants are asked to ignore variability and are often forced to choose one option over others which may not represent how linguistically diverse individuals use language in the real world (Kutlu et al., 2022). Given the well-documented sampling bias (Godfroid & Adringa, 2023), what our theories can account for represents only a fraction of the world’s population. Recognizing the limitations of our theories is integral as, across different scientific fields, a major concern has been the lack of reproducibility and replicability. These discussions highlight the need for more ecological approaches and the centralization of variability. Therefore, as researchers, we must reevaluate our positionalities. Through such reflexivity practices, we can attain more objectivity.  

In this presentation, I will first present data documenting the lack of reflexivity on speakerhood in multilingualism research through the analysis of published research and the concepts that emerge around speakerhood. Next, I will show how the lack of reflexivity brings forward deficiency perspectives to multilingualism, as well as how deficiency views are seen as the canonical approach in examinations of linguistic diversity and are essentialized as more objective measures in research.  Finally, I will provide an example of challenging the deficit views through reflexivity and the systematic dismantling of fundamental quantitative methods used in second language research through examples of different statistical approaches. 
With this, I aim to problematize current approaches in second language acquisition research that limit equitable and just research approaches and highlight the importance of reflexivity in language science research. 

Statistical significance and pedagogical implications: How much can researchers talk about classroom teaching?

Masatoshi Sato, Universidad Andres Bello

An implied or indirect objective of instructed second language acquisition (ISLA) research is to inform second language teachers of evidence-based pedagogical techniques and materials. Quantitative ISLA studies, often based on a classroom-based quasi-experimental design, endeavor to achieve this objective primarily by transforming statistically significant results to teaching practices. However, the extent to which the results can or should be generalized to a different educational context may require the researcher’s reflexivity. In this talk, as a researcher who positions himself as a mediator between research and practice, I discuss the connection between quantitative ISLA research and its pedagogical implications, for several fronts. First, I will discuss researchers’ intention and hope when conducting classroom research. Second, I will zero in on potential internal validity issues by presenting common methodological challenges in classroom-based quasi-experimental research. Third, I will focus on context (e.g., participant characteristics, target language, socio-educational and socio-cultural environment) as an important consideration when ISLA research is transformed to classroom practices. In conclusion, I will call for ISLA researchers’ reflexivity related to: (a) the value of statistically non-significant findings; (b) the balance between ecological validity and other types of validity; (c) the confidence for pedagogical implications based on a single study; (d) the target beneficiaries including teachers; and (e) the extra effort for communicating with practitioners as per their original objective to inform classroom teaching.

Reflexivity in grant-funded educational policy research on multilingualism: Navigating complex researcher-policymaker relationships

Koen Van Gorp, Michigan State University

The recent call to include reflexivity in quantitative research poses particular challenges in grant-related research where external stakeholders and policymakers set the agenda. Although both applied linguists and policymakers share an interest in promoting evidence-based or evidence-informed policies, applied linguistics and the policy arena operate under different paradigms (Burkhardt & Schoenfeld, 2003). While policymakers can initiate a research call, the resulting research findings are just one type of policy-relevant fact in the policy decision-making process (Strassheim & Kettunen, 2014). Alternatively, researchers can initiate studies that might align with local policy agendas and potentially impact educational policy (Slembrouck et al., 2018). Needless to say, the relationship between researchers and policymakers is complex and often contested (Parkhurst, 2017).
In this paper, I reflect on my own positionality as a researcher operating within a center for language and education reliant on external grant funding for its research. As a researcher and later as part of the leadership team of the center, ethical, policy, and practical considerations have been important drivers in the research choices made to support multilingual learners in Flemish primary education. Using the Home Language in Education project within the city of Ghent as a case (Slembrouck et al., 2018; Sierens et al., 2022), I will illustrate how operating within existing language ideologies (mostly monolingual) and local and national political agendas (balancing between valuing multilingualism and promoting the language of schooling) impacts a researcher’s choices, values, and beliefs. Positive research-based policy impact, especially in the field of multilingualism, is not guaranteed as study findings are often experienced by policymakers as “uncomfortable knowledge” (Rayner, 2012). Interaction with policymakers may provide reality checks as well as opportunities for policy co-construction (Cairney & Oliver, 2017), and will inevitably push researchers to rethink or reframe their positionality and research choices.

>>>Back to AAAL 2025 Invited Colloquia