#research-methodology

#research-methodology

Generative AI is now incorporated into the workflow for many scholars across many disciplines, but the broader scientific community would benefit from taking stock of how this technology could truly benefit our work and how it might distract. We hope the symposium can provide clarity.

The researchers think it is fine to tell you only about the time it took each participant to get out of the box. After all, it is a study of box-escaping skill. Often, there is a highly relevant context to the story that is not mentioned. In my hypothetical example, it looks like this: The single person is in the box on the left. The door is shut, and there are boulders in front of it. The top of the box is taped shut.

Transgender women possess no physical advantage over those who were born female, a controversial study claims. Researchers from the University of Sao Paolo in Brazil analysed the strength, fitness and body composition of transgender people undergoing hormonal treatment compared to cisgender individuals. Their findings suggest that while transgender women still have greater muscle mass following hormone therapy, their physical fitness is 'comparable' to other females.

The other researcher suggested that the methodology we use should, to a large extent, be dictated by our epistemological philosophy. For example, are you a positivist, interpretivist, a hypothetico-deductivist, a post-positivist or some other stance appearing on the list of epistemological perspectives? I imagine many readers of this blog, like myself, will be surprised by this stance. Since day one of my research methods training, I've been taught that it's the research question that should dictate your methodology...

AI is not just another research tool; it is redefining what research is, how it is done and what counts as an original contribution. Universities are mostly unprepared for the scale of disruption, with few having comprehensive governance strategies. Many academics remain focused on the failings of early generative AI tools, such as hallucinations (confidently stated but false information), inconsistencies and superficial responses. But AI models that were clumsy in 2023 are becoming increasingly fluent and accurate.

Its eclectic nature. I am trained in a style of research that places value in the faithful documentation and analysis of the actual practices of experts in certain technical fields. Sometimes this line of work runs the risk of being too removed from the immediate advancement of such fields. (The rather polemical analogy is "as useful as ornithology to birds," but I'd like to think of it as a kind of ethnography of certain specialized expertise.)

Researchers Deng et al. (2025) recently asked: Does ChatGPT enhance student learning? They conducted a meta-analysis including 69 studies published from 2022 to 2024. Looking at the overall effects across all studies, they concluded that ChatGPT enhances academic performance, affective-emotional states, and higher-order thinking propensities, while also reducing mental effort. However, Weidlich et al. (2025) identified several flaws in the meta-analysis.

In comparing the replication data analysis to the original study, we noted a significant decrease in effectiveness, from close to 60% to nearly 40%, indicating potential variability in treatment outcomes.

The grounded theory methodology is suitable for building theory in fields where little is known, making it ideal for investigating AB testing in BPI.

To enhance construct validity, we implemented strategies such as pilot experiments for data labelling agreements and consensus involvement to mitigate personal bias.

In evaluating LLM-generated mutations, we designed metrics that encompass cost, usability, and behavior, recognizing that higher mutation scores don't guarantee higher quality.