A Research-Based Checklist for Development and Critique of STEM Instructional Videos
Video has become enormously popular as an instructional medium. Although a large body of education research is available to inform the selection and design of instructional videos, this research is dispersed across journals, disciplinary traditions and STEM fields, and the practical lessons are not always explicit. To guide the development and critique of science and mathematics instructional videos, our interdisciplinary team, with members from mathematics, physics, chemistry and biology, developed an instrument by synthesizing the relevant education literature and translating it into recommendations for practice. The user-friendly instrument, which provides a conceptual foundation and evaluation framework for designers of educational videos, is a 12-item checklist grouped into three categories.
Content and sequencing
- Concepts
- Logic
- Story
- Language
Cognitive supports
- Visualizations
- Signals
- Synchronization
- Segmentation
- Streamlining
Affective considerations
- Relevance
- Rapport
- Accessibility
Video Design Guide
Concepts
Students come to learning situations with prior knowledge, which is often a combination of scientifically normative ideas and misconceptions (NRC, 2000). Although debate is ongoing about the exact form of students’ prior ideas, there is strong agreement that they need to be considered in the process of curriculum design, and that the goal should be to help students integrate their prior knowledge with the new information taught (diSessa, 2014; Duit & Treagust, 2012). Videos can help students make links between prior knowledge and new concepts (Mitra, Lewin-Jones, Barrett & Williamson, 2010). When research has identified commonly held misconceptions about a STEM topic, learners benefit from instructional videos that specifically address those misconceptions (Muller, Bewes, Sharma & Reimann, 2007).
Logic
Research on learning progressions or learning trajectories has combined empirical studies of how students’ knowledge of science or mathematics topics develops over time with content analyses of the disciplines themselves, with the aim of informing appropriate curriculum sequences (Duschl, Maeng & Sezen, 2011). Although any individual video can only address a narrow portion of a learning progression, the content of the video should still be consistent with overall learning progression for a topic, based on grade or age. Gaps in logic are common in textbooks (Seethaler, Czworkowski & Lynn, 2017). Videos should avoid them, and may be especially useful if they address gaps in traditional curriculum materials
Story
Narrative formats can increase comprehension and engagement (Dahlstrom, 2014). This suggests that video designers should consider incorporating narrative, but it would be overly limiting to suggest that all educational videos should follow one particular format, because various formats of videos can promote learning (Guo, Kim & Rubin, 2014). Nevertheless, ample research indicates that students are more likely to learn when they are engaged with a specific question or problem (Bybee, 2014). Video designers should endeavor to construct an intellectual story beginning with a puzzle that is resolved by the end of the video. Such a video can be brief while still encompassing a story arc.
Language
With respect to language, research supports “personalization” and “content first.” Mayer’s personalization principle is supported by research that shows students learn better from multimedia lessons when the narrative is conversational (i.e. using first and second person and directing comments at the viewer) rather than formal in style (Mayer, Fennell, Farmer & Campbell, 2004; Kartal, 2010). Content-first approaches to teaching are supported by research demonstrating that academic language is a significant hurdle in science learning and that students perform better when scientific concepts are introduced before the relevant technical terms are presented (Brown & Ryoo, 2008; McDonnell, Barker & Wieman, 2016).
Visualizations
Many aspects of our world are too large or too small to be seen with the naked eye and many processes occur on too long or short a timescale for us to observe them directly. Science visualizations have a long history in science education in helping students understand microscopic, submicroscopic and astronomical structures, physical phenomena, and complex systems (Eick & King, 2012; Linn, 2003). Dynamic visualizations such as animations can support learning by portraying things that static visualizations cannot (Tversky, Morrison & Betrancourt, 2002). Dynamic visualizations can also address specific misconceptions and close gender gaps in understanding (Yezierski & Birk, 2006). Visual attention limits what can be learned from an animation, particularly when changes are happening simultaneously in different parts of the animation (Hegarty, 2005). Visualizations should therefore take advantage of the affordances of the video medium to make science visible to students, while taking care to avoid overwhelming visual attention.
Signals
Moving between the macroscopic and microscopic or submicroscopic world, and between representations (in the form of words, graphs and symbols), is a significant challenge in science learning (Johnstone 1991, 1993). Novice learners may not perceive key features of representations without assistance, and may focus on design elements instead of the underlying concepts (Cook, 2006; Tasker, 2016). For learners to create referential connections between representations, instructors and instructional materials should make these links explicit through the use of cues, including arrows, highlights and verbal guidance. Variations, such as in color, size or motion cue learners to focus on the varying features, which means designers must take care to avoid inadvertently emphasizing unimportant features (Bussey & Orgill, 2015). For example, video developers should take heed of studies of traditional curriculum materials, which have demonstrated that use of (static) arrows is often inconsistent and confusing to students and that students need support to interpret them (Wright, Cardenas, Liang & Newman, 2017).
Synchronization
People learn better from words and pictures than from words alone. Words and graphics should be presented concurrently, rather than successively, to help learners build connections between them to be stored in long-term memory (Moreno, 2006). However, the visual channel can become overloaded when both words and images need to be processed through it. Thus presenting graphics with narration supports learning better than presenting the words as written text, as spoken words can be processed through the auditory channel while the visual channel is free to process the images (Mayer & Moreno, 2003; Moreno & Mayer, 1999). Likewise, multimedia formats that present slides and a separate window showing the speaker can split viewer’s visual attention and interfere with learning; videos should only require one visual focal point at a time (Chen & Wu, 2015). That being said, for accessibility, the use of captioning should be available.
Segmentation
Information should be divided into temporal segments that learners can digest one at a time before moving on (Mayer & Moreno, 2003). An often-cited study of student engagement with MOOCs recommends that educational videos be no more than 6 minutes in length (Guo, Kim & Rubin, 2014). MOOCs, however, are a unique educational environment because enrollment is typically not for credit. In a study of problem-solving videos in undergraduate chemistry courses, in which the videos averaged 10 minutes in length (range 2 to 28), engagement with the videos was high, feedback overwhelmingly positive, and nearly ten times as many students (32.5%) wanted videos with additional problems versus recommended (3.9%) decreasing video length (Richards-Babb, Curtis, Smith & Xu, 2014). A one-size-fits-all recommendation on video length is thus unwarranted; instead, video length and pacing should be carefully contingent on the amount and complexity of information covered.
Streamlining
Cognitive load theory is a key theoretical underpinning for the design of multimedia (Sweller & Chandler, 1994). The cognitive load imposed by a video should be germane, and extraneous processing--that which does not serve instructional objectives--should be minimized (Mayer & Moreno, 2003). For example, irrelevant sound or music can impede learning (Moreno & Mayer, 2000). Overly long explanations, and conflicts between written and spoken words that occur when text-heavy slides compete with narration, are also detrimental to learning (Mayer, Bove, Bryman, Mars & Tapangco, 1996; Moreno, 2006). Thus, video content and decoration should be streamlined as much as possible, with exceptions noted in the “Affective Considerations” section.
Relevance
A large body of research supports the importance of “relevance” as a motivational factor in STEM education, where relevance can be summarized in three overlapping dimensions: individual, societal and professional (Stuckey, Hofstein, Mamlok-Naaman and Eilks, 2013). Designers of educational videos should strive to select content and scenarios that appeal to students’ curiosity and interests, and help prepare them for civic life and future careers. Students’ motivation is also influenced by multimedia design features, such as color and appealing graphics (Mayer & Estrella, 2014; Plass, Heidig, Hayward, Homer & Um, 2014). When designs induce positive emotions, learners’ intrinsic motivation is enhanced to continue working with the materials (Heidig, Müller & Reichelt, 2015). To avoid cognitive overload, emotional design elements must relate to the essential content of the lesson (Mayer, 2014).
Rapport
Vygotsky’s Zone of Proximal Development refers to the difference between what a learner can do alone and what the learner can do with support from others (Vygotsky, 1978). With respect to video, the social support comes in the form of vicarious learning--being able to “listen in” on peers’ discussions with one another or with a tutor. Dialogue in video, despite the extra cognitive load it can impose, is at least as effective as expository instructional formats with respect to learning gains (Cox, McKendree, Tobin, Lee & Mayes, 1999). Dialogue can have additional affective benefits. Seeing their ideas represented by peers can help students feel part of a community of learners, even to the degree of treating characters in videos as quasi-collaborators (Lobato & Walker, 2019). Students in videos should be empowered to ask questions and make mistakes (Muller, Sharma, Reimann, 2008). Although research to date does not support the hypothesis that students necessarily learn better from instructors or models who are like them with respect to age, gender or ethnicity, we would argue that being inclusive and avoiding stereotypes is an important aspect of respecting and empowering learners (Hoogerheide, Loyens & Gog, 2016; Hoogerheide, van Wermeskerken, Loyens & Gog, 2016; Moreno & Flowerday, 2006; Liew, Tan & Jayothisa, 2013).
Accessibility
On the one hand, learner engagement with multimedia is not contingent on high production value (Guo, Kim & Rubin, 2014). On the other hand, poor video quality negatively influences audience perceptions of scientific research and researchers (Newman & Schwartz, 2018). Quality, therefore, must be adequate to support learning. Similarly, to make educational videos useable for as diverse a population as possible, design should incorporate the Principles of Universal Design, a set of seven broad principles with finer-grained guidelines to make products and environments widely accessible and useable (Center for Universal Design, 1997). The guidelines relevant to video design include: make the design appealing to all users; eliminate unnecessary complexity; be consistent with user expectations and intuitions; maximize legibility of essential information; arrange information according to its importance; and provide compatibility with devices used by people with sensory limitations (Story, 2001). For example, select the contrasting colors and shades of signals so that they can also be distinguished by those with colorblindness, ensure that closed captions will not cover important content, and incorporate pauses in the video soundtrack to leave time for audio descriptions for the blind. A distinct set of accessibility design guidelines, Universal Design for Learning Guidelines (with an accompanying a 37-item UDL Scan Tool), can help instructors assess the accessibility of their overall curriculum (Smith & Harvey, 2014). The UDL Scan Tool’s exclusive focus on the need to provide learners with options (e.g. for expressive skills, self-regulation, executive function), however, makes it less relevant for evaluating individual videos.
Seethaler, S., Burgasser, A. J., Bussey, T. J., Eggers, J., Lo, S. M., Rabin, J. M., Stevens, L. & Weizman, H. (2020). A research-based checklist for development and critique of STEM instructional videos. Journal of College Science Teaching, 50(1), 21-27.
