--- title: "Evaluating AI-generated questions: A mixed-methods analysis using question data and student perceptions" description: Automatic Question Generation --- < Back to Publications # Evaluating AI-generated questions: A mixed-methods analysis using question data and student perceptions ## Abstract Advances in artificial intelligence (AI) have made it possible to generate courseware and formative practice questions from textbooks. Courseware applies a learn by doing approach by integrating formative practice with text, a method proven to increase learning gains for students. By using AI for automatic question generation, the learn by doing method of courseware can be made available for nearly any textbook subject. As the generated questions are a primary learning feature in this environment, it is necessary to ensure they function as well for students as those written by humans. In this paper, we will use student data from an AI-generated Psychology courseware used in an online course at the University of Central Florida. The courseware has both generated questions and human-authored questions, allowing for a unique comparison of question engagement, difficulty, and persistence using student data from a natural learning context. The evaluation of quality metrics is critical in automatic question generation research, yet on its own is not comprehensive of students’ experience. Student perception is a meaningful qualitative metric, as student perceptions can inform behavior and decisions. Therefore, student perceptions of the courseware and questions were also solicited via survey. Combining question data analysis with student perception feedback gives a more comprehensive evaluation of the quality of AI-generated questions used in a natural learning context.  [Read Publication](https://doi.org/10.1007/978-3-031-11644-5_28) ##### Date July 27, 2022 ##### Authors - Rachel Van Campenhout - Martha Hubertz - Benny G. Johnson ##### Research Areas [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) ##### Venue Artificial Intelligence in Education (AIED) ##### Citation Van Campenhout, R., Hubertz, M., & Johnson, B. G. (2022). Evaluating AI-generated questions: A mixed-methods analysis using question data and student perceptions. *Artificial Intelligence in Education. AIED 2022. Lecture Notes in Computer Science, Vol. 13355*, 344–353. Springer. [https://doi.org/10.1007/978-3-031-11644-5_28](https://doi.org/10.1007/978-3-031-11644-5_28) ##### Share [![](https://research.vitalsource.com/hubfs/2021_Research/Social%20-%20Facebook.svg)](https://www.facebook.com/sharer/sharer.php?u=https://research.vitalsource.com/publications/evaluating-ai-generated-questions-a-mixed-methods-analysis-using-question-data-and-student-perceptions) [![](https://research.vitalsource.com/hubfs/2021_Research/Social%20-%20LinkedIn.svg)](https://www.linkedin.com/shareArticle?mini=true&url=https://research.vitalsource.com/publications/evaluating-ai-generated-questions-a-mixed-methods-analysis-using-question-data-and-student-perceptions) [![](https://research.vitalsource.com/hubfs/2021_Research/Social%20-%20Twitter.svg)](https://twitter.com/intent/tweet?url=https://research.vitalsource.com/publications/evaluating-ai-generated-questions-a-mixed-methods-analysis-using-question-data-and-student-perceptions&text=) ### Related Publications #### Search our catalog of recent publications authored by our team. **2026** ##### [Extending an automatic question generation pipeline with LLM-based free-response tasks: An analysis of performance metrics using student data](https://research.vitalsource.com/publications/extending-an-automatic-question-generation-pipeline-with-llm-based-free-response-tasks-an-analysis-of-performance-metrics-using-student-data) **CSEDU 2026 – CSEDU 18th International Conference on Computer Supported Education** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Jeffrey S. Dittel, Bill Jerome, Benny G. Johnson (2026) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Teaching & Learning](https://research.vitalsource.com/publications/tag/teaching-learning) **2026** ##### [Student attitudes and perceptions of AI and AI-generated practice in the classroom](https://research.vitalsource.com/publications/student-attitudes-and-perceptions-of-ai-and-ai-generated-practice-in-the-classroom) **The IAFOR International Conference on Education – Hawaii 2026** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Michelle Clark (2026) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Teaching & Learning](https://research.vitalsource.com/publications/tag/teaching-learning) **2025** ##### [AI-generated formative practice and feedback: Performance benchmarks and applications in higher education](https://research.vitalsource.com/publications/ai-generated-formative-practice-and-feedback-performance-benchmarks-and-applications-in-higher-education) **Artificial Intelligence in Measurement and Education Conference (AIME-CON) ** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Michelle Clark, Jeffrey S. Dittel, Bill Jerome, Nick Brown, Benny G. Johnson (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Feedback](https://research.vitalsource.com/publications/tag/feedback) **2025** ##### [Improving automatically generated fill-in-the-blank answer selection with an LLM-based agreement filter](https://research.vitalsource.com/publications/improving-automatically-generated-fill-in-the-blank-answer-selection-with-an-llm-based-agreement-filter) **Educational Data Mining 2025 Workshop on Causal Inference** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Benny G. Johnson, Jeffrey S. Dittel, Rachel Van Campenhout (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) **2025** ##### [Scaling effective characteristics of ITSs: A preliminary analysis of LLM-based personalized feedback](https://research.vitalsource.com/publications/scaling-effective-characteristics-of-itss-a-preliminary-analysis-of-llm-based-personalized-feedback) **International Conference on Intelligent Tutoring Systems** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Jeffrey S. Dittel, Benny G. Johnson (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Feedback](https://research.vitalsource.com/publications/tag/feedback) **2025** ##### [Open-ended questions need personalized feedback: Analyzing LLM-enabled features with student data](https://research.vitalsource.com/publications/open-ended-questions-need-personalized-feedback-analyzing-llm-enabled-features-with-student-data) **Second Workshop on Automated Evaluation of Learning and Assessment Content at the 26th International Conference on Artificial Intelligence in Education** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Jeffrey S. Dittel, Bill Jerome, Michelle W. Clark, Benny G. Johnson (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Feedback](https://research.vitalsource.com/publications/tag/feedback) [See all publications](https://research.vitalsource.com/publications)