NoteLink: A Point-and-Shoot Linking Interface between Students' Handwritten Notebooks and Instructional Videos

When learning from instructional videos, students frequently take handwritten notes to improve recall and comprehension. When reviewing their notes, it can be difficult to return to the corresponding part of the video. In this paper, we present NoteLink, a mobile application that allows students to take pictures of their notes to re-find and play relevant videos on their smartphone or tablet. Our study followed four phases. In Phase I, we identified the characteristics of students' notes by analyzing 10 engineering students' handwritten notes taken while watching instructional videos. We found: 1) students' notes are comprised of four content types: text, formula, drawing, and a hybrid of two or more types, 2) at least 75% of the notes, regardless of content type, manifest some degree of verbatim overlap with the corresponding video content, and 3) videos are referenced at three scales of temporal granularity: point, interval, and whole video. In Phase II, we designed a prototype mobile application, NoteLink, that retrieves instructional videos that are similar to students' notes. In Phase III, we ran a usability study with 12 engineering students to evaluate their preferences for the temporal granularity of retrieved videos and how search results are displayed. Students reported a preference for matches at the interval temporal granularity. Interviews with participants suggest that NoteLink-like tools for re-finding instructional videos are useful. In Phase IV, we evaluated the retrieval accuracy of NoteLink using the data collected in Phase I. The overall accuracy was 78%, and 98% for textual notes. We also provide design recommendations for optimizing NoteLink.