ChatLTV
A Slack-embedded retrieval-augmented assistant trained on the course's own materials, available to all 250 enrolled students. It is the only HBS tutorbot with published usage data.2
Corpus: 50-plus HBS cases, teaching notes, two books, slide decks, Excel models, video transcripts with a glossary, nineteen years of the instructor's blog posts, and three years of prior course Slack Q&A.
Stack: GPT-4 accessed through Azure OpenAI, chosen so Harvard's copyrighted case material would not be used for model retraining; Pinecone for vector storage; Langchain as middleware. Roughly 8,000 lines of backend code, of which 800 were the RAG layer and 900 content indexing, plus 9,000 lines for a content management system. Two to three person-months of development in late spring and summer 2023. Bussgang notes it would be substantially smaller if built today.
Testing: approximately 500 hand-written test questions evaluated manually, plus automated scoring against ground-truth answers, used to refine prompts and content indexing.
Retrieval prompt as published:
Testing revealed that students wanted course logistics as much as case content, so a separate "Course Admin" corpus covering grading rubrics, writing assignments, due dates and office hours was created and prioritized in retrieval.
Question types: clarifying and evaluative questions about cases, analysis, acronyms, and administrative matters — case summaries, acronym definitions, assignment deadlines, office-hour scheduling, and methodology checks. Usage peaked between 10pm and 2am. Nearly 40 percent of users rated quality a 4 or 5.
What the query logs showed
Bussgang reviewed queries each morning through the content management system and reports three cases. A student who participated little in class had submitted numerous thoughtful queries; on that basis Bussgang cold-called him, and reports he "crushed the opening." A non-native English speaker repeatedly asked for acronym definitions; Bussgang had ChatLTV generate a list of the top fifteen course acronyms with definitions, after which her questions became more advanced. A student who was a new parent was preparing cases several days early; asked about it, she explained that an unpredictable sleep schedule meant she needed to refresh the key facts closer to class.
Bussgang states that his concern that students would use the bot as a crutch did not materialize, that the quality of in-class case conversation was excellent, and that prep-level questions moving to the bot let him push students on assumptions, methodology and strategic implications instead.
He also reports that the LLM required considerable prompt engineering, that a large existing corpus was essential to the result, and that he had to instruct a companion feedback tool to be more critical, writing: "LLMs are way softer than HBS professors."