Developing and Evaluating the Use of ChatGPT as a Screening Tool for Nurses Conducting Structured Literature Reviews: Proof of Concept Study Results

Aim

To examine the feasibility of using a large language model (LLM) as a screening tool during structured literature reviews to facilitate evidence-based practice.

Design

A proof-of-concept study.

Methods

This paper outlines an innovative method of abstract screening using ChatGPT and computer coding for large scale, effective and efficient abstract screening. The authors, new to ChatGPT and computer coding, used online education and ChatGPT to upskill. The method was empirically tested using 400 abstracts relating to public involvement in nursing education from four different databases (CINAHL, Scopus, ERIC and MEDLINE), using four versions of ChatGPT. Results were compared with a human nursing researcher and reported using the CONSORT 2010 extension for pilot and feasibility trials checklist.

Results

ChatGPT-3.5 Turbo was most effective for rapid screening and had a broad inclusionary approach with a false-negative rate lower than the human researcher. More recent versions of ChatGPT-4, 4 Turbo, and 4 omni were less effective and had a higher number of false negatives compared to ChatGPT-3.5 Turbo and the human researcher. These more recent versions of ChatGPT did not appear to appreciate the nuance and complexities of concepts that underpin nursing practice.

Conclusion

LLMs can be useful in reducing the time nurses spend screening research abstracts without compromising on literature review quality, indicating the potential for expedited synthesis of research evidence to bridge the research–practice gap. However, the benefits of using LLMs can only be realised if nurses actively engage with LLMs, explore LLMs' capabilities to address complex nursing issues, and report on their findings.

Implications for the Professional and/or Patient Care

Nurses need to engage with LLMs to explore their capabilities and suitability for nursing purposes.

Patient or Public Contribution

No patient or public contribution.