--- title: "Toward effective courseware at scale: Investigating automatically generated questions as formative practice" description: Learning @ Scale 2021 --- < Back to Publications # Toward effective courseware at scale: Investigating automatically generated questions as formative practice ## Abstract Courseware is a comprehensive learning environment that engages students in a learning by doing approach while also giving instructors data-driven insights on their class, providing a scalable solution for many instructional models. However, courseware-and the volume of formative questions required to make it effective-is time-consuming and expensive to create. By using artificial intelligence for automatic question generation, we can reduce the time and cost of developing formative questions in courseware. However, it is critical that automatically generated (AG) questions have a level of quality on par with human-authored (HA) questions in order to be confident in their usage at scale. Therefore, our research question is: are student interactions with AG questions equivalent to HA questions with respect to engagement, difficulty, and persistence metrics? This paper evaluates data for AG and HA questions that students used as formative practice in their university Communication course. Analysis of AG and HA questions shows that our first generation of AG questions perform equally well as HA questions in multiple important respects. [Read Publication](https://dl.acm.org/doi/10.1145/3430895.3460162) [View Data Set](https://github.com/vitalsource/data/tree/f92f2d87acbc69c1c654717e9ea971e300cd0465/l%40s-2021) ##### Date June 8, 2021 ##### Authors - Rachel Van Campenhout - Nick Brown - Bill Jerome - Jeffrey S. Dittel - Benny G. Johnson ##### Research Areas [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) ##### Venue Eighth ACM Conference on Learning @ Scale ##### Citation Van Campenhout, R., Brown, N., Jerome, B., Dittel, J. S., & Johnson, B. G. (2021). Toward effective courseware at scale: Investigating automatically generated questions as formative practice. *Learning @ Scale*, 295–298. ACM. [https://doi.org/10.1145/3430895.3460162](https://doi.org/10.1145/3430895.3460162) ##### Share [![](https://research.vitalsource.com/hubfs/2021_Research/Social%20-%20Facebook.svg)](https://www.facebook.com/sharer/sharer.php?u=https://research.vitalsource.com/publications/toward-effective-courseware-at-scale-investigating-automatically-generated-questions-as-formative-practice) [![](https://research.vitalsource.com/hubfs/2021_Research/Social%20-%20LinkedIn.svg)](https://www.linkedin.com/shareArticle?mini=true&url=https://research.vitalsource.com/publications/toward-effective-courseware-at-scale-investigating-automatically-generated-questions-as-formative-practice) [![](https://research.vitalsource.com/hubfs/2021_Research/Social%20-%20Twitter.svg)](https://twitter.com/intent/tweet?url=https://research.vitalsource.com/publications/toward-effective-courseware-at-scale-investigating-automatically-generated-questions-as-formative-practice&text=) ### Related Publications #### Search our catalog of recent publications authored by our team. **2026** ##### [Extending an automatic question generation pipeline with LLM-based free-response tasks: An analysis of performance metrics using student data](https://research.vitalsource.com/publications/extending-an-automatic-question-generation-pipeline-with-llm-based-free-response-tasks-an-analysis-of-performance-metrics-using-student-data) **CSEDU 2026 – CSEDU 18th International Conference on Computer Supported Education** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Jeffrey S. Dittel, Bill Jerome, Benny G. Johnson (2026) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Teaching & Learning](https://research.vitalsource.com/publications/tag/teaching-learning) **2026** ##### [Student attitudes and perceptions of AI and AI-generated practice in the classroom](https://research.vitalsource.com/publications/student-attitudes-and-perceptions-of-ai-and-ai-generated-practice-in-the-classroom) **The IAFOR International Conference on Education – Hawaii 2026** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Michelle Clark (2026) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Teaching & Learning](https://research.vitalsource.com/publications/tag/teaching-learning) **2025** ##### [AI-generated formative practice and feedback: Performance benchmarks and applications in higher education](https://research.vitalsource.com/publications/ai-generated-formative-practice-and-feedback-performance-benchmarks-and-applications-in-higher-education) **Artificial Intelligence in Measurement and Education Conference (AIME-CON) ** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Michelle Clark, Jeffrey S. Dittel, Bill Jerome, Nick Brown, Benny G. Johnson (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Feedback](https://research.vitalsource.com/publications/tag/feedback) **2025** ##### [Improving automatically generated fill-in-the-blank answer selection with an LLM-based agreement filter](https://research.vitalsource.com/publications/improving-automatically-generated-fill-in-the-blank-answer-selection-with-an-llm-based-agreement-filter) **Educational Data Mining 2025 Workshop on Causal Inference** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Benny G. Johnson, Jeffrey S. Dittel, Rachel Van Campenhout (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) **2025** ##### [Scaling effective characteristics of ITSs: A preliminary analysis of LLM-based personalized feedback](https://research.vitalsource.com/publications/scaling-effective-characteristics-of-itss-a-preliminary-analysis-of-llm-based-personalized-feedback) **International Conference on Intelligent Tutoring Systems** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Jeffrey S. Dittel, Benny G. Johnson (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Feedback](https://research.vitalsource.com/publications/tag/feedback) **2025** ##### [Open-ended questions need personalized feedback: Analyzing LLM-enabled features with student data](https://research.vitalsource.com/publications/open-ended-questions-need-personalized-feedback-analyzing-llm-enabled-features-with-student-data) **Second Workshop on Automated Evaluation of Learning and Assessment Content at the 26th International Conference on Artificial Intelligence in Education** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Jeffrey S. Dittel, Bill Jerome, Michelle W. Clark, Benny G. Johnson (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Feedback](https://research.vitalsource.com/publications/tag/feedback) [See all publications](https://research.vitalsource.com/publications)