In the summer of 2017, Minerva reached another institutional milestone with its first graduating class of master’s students. In order to earn the Master of Science in Decision Analysis, each student conducted four months of study on a real-world challenge of their choosing and, using the skills they acquired throughout the program, developed a master’s thesis to apply their learning in a practical way.
For his thesis, Zhivko Chobanov aimed to understand the extent to which online providers of self-study materials used established best practices from the science of learning. For example, how much do online language learning tools, such as DuoLingo, take advantage of the fact that optimal learning occurs when challenges are spread out over time — known as “spaced practice” — rather than bundled together in one high-stakes test? Chobanov also wanted to develop a tool to empower others to evaluate these online providers themselves. His most interesting application of the program learning objectives, however, pertained not to the subject of his thesis, but to the way he managed the project of writing the thesis itself.
Chobanov applied a collection of methods taught in our three core courses that can be used to help deal with risks associated with managing a complex project. For example, we teach the practice of “broad framing,” which reminds decision makers to stay true to their goals by considering holistic aggregations of decisions rather than isolated choices.
Chobanov applied “broad framing” over a period of two months while diligently reviewing more than 100 papers on education. With no deadlines and a seemingly endless amount of articles to get through, there were times this undertaking seemed daunting, insignificant, and less beneficial than engaging in an activity that offered a more immediate reward. Keeping in mind, however, the importance of this task in the context of his whole project, Chobanov decided to not attend to the temporary discomforts associated with his efforts, and instead to the broader benefit that those efforts would yield at the end of the summer: a graduate degree. To remain disciplined, he temporarily decreased his communication with friends who were on summer vacation, increased his communication with classmates who were also working on their theses, and explicitly asked those around him to repeatedly question him about his progress.
We also teach students to avoid the pitfalls of temporal discounting, in which future outcomes are given short shrift. A classic example is a choice: would you prefer a $10 gift today or a $100 gift five years from now? Though he first learned this concept in classes about genetically modified organisms and nuclear weapons, Chobanov applied discounting to an entirely new domain: his work on the thesis project itself.
For example, recognizing that temporal discounting can explain the fallacious tendency to believe that one’s future self would somehow be more productive than one’s current self, and that tasks tend to expand to fill the time available to them, Chobanov wrote a complete draft of each section as early as possible, rather than waiting until he felt that he was completely ready to write them. For instance, he drafted the first paragraphs on “controlled practice” and “elaborative interrogation” early on to use for making predictions about the word count and the time needed to write about each educational technique he planned to cover. He also elected to write in paragraphs rather than bullets, developed and defended original arguments as needed, and included segues to surrounding material. This process proved to Chobanov that he had little time for breaks, even though he had a month and a half remaining before his project was due.
Aware that narrow framing and temporal discounting could lead to his getting lost in tangential, inessential work, Chobanov made a strict catalog of the tasks that were truly necessary for the completion of the overall project, and hewed closely to it.
For example, he allotted a set amount of time to researching 12 distinct teaching techniques — techniques such as spaced practice, scaffolding, and worked examples. When that time had elapsed, he moved on to the next section of this project, even though he could have spent arbitrarily more time on this one, simply because the literature on the topic is so extensive. Here, he invoked the 80/20 rule, another of the decision-making heuristics we teach in Advanced Formal Analyses. The 80/20 rule says that most of the results (80%) come from a small portion (20%) of the inputs. Chobanov knew that his initial time estimate for the task couldn’t have been terribly inaccurate, and therefore, even doubling the amount of time spent would probably yield only marginal improvement. At that stage in the project, the smart move was to advance to the next step, and fine-tune the end result once a draft was complete.
Chobanov concluded that only some providers of online self-study materials use techniques found in active learning and, those that do, do not do so consistently. Perhaps more importantly and due in no small part to his effective project management, Chobanov succeeded in developing an original framework to evaluate the efficacy of online learning tools with the help of advisors at Minerva and the Massachusetts Institute of Technology (MIT). The resulting framework can be used as a checklist to help self-study services determine the strengths and weaknesses of their products, and provides research-based guidance for how to improve them. Chobanov also leveraged and strengthened his status as a web developer by making the framework publicly available.