Development and validation of ChatGPT literacy scale

ChatGPT revolutionized the landscape of information transitions and creativity. The number of ChatGPT users is increasing at the fastest pace in consumer applications history, which calls for a reliable ChatGPT literacy scale. To that end, first, we conducted a literature review to gain a conceptual understanding of the ChatGPT literacy construct. Next, we conducted a focus group consisting of five experts to brainstorm potential items and the following five factors of ChatGPT literacy: technical proficiency, critical evaluation, communication proficiency, creative application, and ethical competence. The Delphi method with 10 experts was used to validate these items. After collecting expert feedback and evaluation data, we calculated content validity ratios to analyze the items’ content validity. Another focus group interview was conducted with 10 university students as a pilot test. Finally, 822 college students were recruited to evaluate the construct validity of ChatGPT literacy through exploratory and confirmatory factor analysis, resulting in 25 final items. This novel approach is expected to contribute to building a theoretical basis for ChatGPT and large language model studies, providing practical implications for large language model developers, helping to develop related regulations and policies, and creating a healthy digital culture for the use of ChatGPT.