--- title: Iterative improvement of automatically generated practice with the Content Improvement Service description: Adaptive Instructional Systems, HCII 2023 --- < Back to Publications # Iterative improvement of automatically generated practice with the Content Improvement Service ## Abstract The Content Improvement Service (CIS) is a platform-level adaptive system that monitors millions of automatically generated formative practice questions that are available to students as a study feature in thousands of textbooks. The CIS was designed to do what was not possible by humans—use real-time data to monitor question performance at enormous scale and determine if a change is required. The CIS was designed to use multiple types of data and analyses to make its decisions, including question difficulty (mean score) and student feedback (ratings). In this paper, we outline the decisions made by the CIS for both methods. We also show how human investigation of these analyses can identify trends and insights useful to automatic question generation systems. [Read Publication](https://doi.org/10.1007/978-3-031-34735-1_22) ##### Date July 9, 2023 ##### Authors - Bill Jerome - Rachel Van Campenhout - Jeffrey S. Dittel - Rich Benton - Benny G. Johnson ##### Research Areas [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Adaptivity](https://research.vitalsource.com/publications/tag/adaptivity) [Learning Engineering](https://research.vitalsource.com/publications/tag/learning-engineering) ##### Venue Adaptive Instructional Systems, HCII 2023 ##### Citation Jerome, B., Van Campenhout, R., Dittel, J. S., Benton, R., & Johnson, B. G. (2023). Iterative improvement of automatically generated practice with the Content Improvement Service. *Adaptive Instructional Systems. HCII 2023. Lecture Notes in Computer Science, Vol. 14044*, 312–324. Springer. [https://doi.org/10.1007/978-3-031-34735-1_22](https://doi.org/10.1007/978-3-031-34735-1_22) ##### Share [![](https://research.vitalsource.com/hubfs/2021_Research/Social%20-%20Facebook.svg)](https://www.facebook.com/sharer/sharer.php?u=https://research.vitalsource.com/publications/iterative-improvement-of-automatically-generated-practice-with-the-content-improvement-service) [![](https://research.vitalsource.com/hubfs/2021_Research/Social%20-%20LinkedIn.svg)](https://www.linkedin.com/shareArticle?mini=true&url=https://research.vitalsource.com/publications/iterative-improvement-of-automatically-generated-practice-with-the-content-improvement-service) [![](https://research.vitalsource.com/hubfs/2021_Research/Social%20-%20Twitter.svg)](https://twitter.com/intent/tweet?url=https://research.vitalsource.com/publications/iterative-improvement-of-automatically-generated-practice-with-the-content-improvement-service&text=) ### Related Publications #### Search our catalog of recent publications authored by our team. **2026** ##### [Extending an automatic question generation pipeline with LLM-based free-response tasks: An analysis of performance metrics using student data](https://research.vitalsource.com/publications/extending-an-automatic-question-generation-pipeline-with-llm-based-free-response-tasks-an-analysis-of-performance-metrics-using-student-data) **CSEDU 2026 – CSEDU 18th International Conference on Computer Supported Education** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Jeffrey S. Dittel, Bill Jerome, Benny G. Johnson (2026) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Teaching & Learning](https://research.vitalsource.com/publications/tag/teaching-learning) **2026** ##### [Student attitudes and perceptions of AI and AI-generated practice in the classroom](https://research.vitalsource.com/publications/student-attitudes-and-perceptions-of-ai-and-ai-generated-practice-in-the-classroom) **The IAFOR International Conference on Education – Hawaii 2026** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Michelle Clark (2026) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Teaching & Learning](https://research.vitalsource.com/publications/tag/teaching-learning) **2025** ##### [AI-generated formative practice and feedback: Performance benchmarks and applications in higher education](https://research.vitalsource.com/publications/ai-generated-formative-practice-and-feedback-performance-benchmarks-and-applications-in-higher-education) **Artificial Intelligence in Measurement and Education Conference (AIME-CON) ** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Michelle Clark, Jeffrey S. Dittel, Bill Jerome, Nick Brown, Benny G. Johnson (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Feedback](https://research.vitalsource.com/publications/tag/feedback) **2025** ##### [Improving automatically generated fill-in-the-blank answer selection with an LLM-based agreement filter](https://research.vitalsource.com/publications/improving-automatically-generated-fill-in-the-blank-answer-selection-with-an-llm-based-agreement-filter) **Educational Data Mining 2025 Workshop on Causal Inference** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Benny G. Johnson, Jeffrey S. Dittel, Rachel Van Campenhout (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) **2025** ##### [Scaling effective characteristics of ITSs: A preliminary analysis of LLM-based personalized feedback](https://research.vitalsource.com/publications/scaling-effective-characteristics-of-itss-a-preliminary-analysis-of-llm-based-personalized-feedback) **International Conference on Intelligent Tutoring Systems** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Jeffrey S. Dittel, Benny G. Johnson (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Feedback](https://research.vitalsource.com/publications/tag/feedback) **2025** ##### [Open-ended questions need personalized feedback: Analyzing LLM-enabled features with student data](https://research.vitalsource.com/publications/open-ended-questions-need-personalized-feedback-analyzing-llm-enabled-features-with-student-data) **Second Workshop on Automated Evaluation of Learning and Assessment Content at the 26th International Conference on Artificial Intelligence in Education** ![profile](https://research.vitalsource.com/hubfs/2021_Research/Home/profile.svg) Rachel Van Campenhout, Jeffrey S. Dittel, Bill Jerome, Michelle W. Clark, Benny G. Johnson (2025) [Automatic Question Generation](https://research.vitalsource.com/publications/tag/automatic-question-generation) [Generative AI](https://research.vitalsource.com/publications/tag/generative-ai) [Feedback](https://research.vitalsource.com/publications/tag/feedback) [See all publications](https://research.vitalsource.com/publications)