Project: Turning Videos to Course Notes Agentic Pipeline (Part 1)

When is it necessary to chunk and when is it not?

Chunking was considered standard procedure, especially when LLMs were first gaining popularity in the early 2020s. For example, early GPT models had a context window of only ~1k tokens in 2019. Since then, frontier models have pushed context windows from 1k to 1M tokens. As a rough reference point, a general guideline for the number of words on a page is ~500 words, single-spaced, which is about 550–630 tokens, given that one token is 3/4 of a word.

Borrowed from AI Engineering by Chip Huyen (2025)

Frontier models

Frontier models are multimodal language models trained on massive amounts of data. They’re considered the smartest models on the market right now; think OpenAI’s GPT, Anthropic’s Claude, and Google’s Gemini. Below is a comparison of frontier (or near-frontier) models from those providers.

Context window, max output, and knowledge cutoff of frontier (-or near) models.

Each model has its own tokenizer. tiktoken can be used to estimate token counts for OpenAI models, but as far as I know, there isn’t an equivalent for Anthropic and Google models. Token counts for Claude and Gemini are computed during an API call and can be retrieved via metadata, but they can’t be reliably estimated in advance the way OpenAI’s models can with tiktoken.

To get started, I’m using Module 1 to design an effective prompt. Module 1 contains 10 transcripts, and I calculated the word and token counts for each transcript.

Given that I’m still in the testing phase, I’m using gpt-5-mini to keep costs low. The context window for this model is 400k max tokens.

The total token count for Module 1 is 26,166. So even if I pass the entire module to GPT, I’ll still have plenty of room left given the upper limit of the context window. Keeping this in mind, I concatenated all transcripts from Module 1 and passed them to the LLM.

Iteration 1

In the first iteration, I asked the LLM to generate notes with the following structure:

# Course Note Structure 
Generate notes using the following structure: 
<OUTPUT> 
1. Title 
2. Overview 
3. Key Concepts 
4. Detailed Explanations 
5. Examples 
6. Mermaid Diagram 
7. Common Pitfalls 
8. Best Practices 
9. Key Takeaways </OUTPUT>​

What I found: even though I asked for structured course notes, it produced summaries instead.

My interpretation is that although we’re given a generous context window, not all content in a long prompt receives the same attention. Chip Huyen suggests that, depending on the model, some pay more attention to the beginning of the prompt while others pay more attention to the end. This reinforces that experimentation is critical when developing LLM and AI agent pipelines. We need to get to know the models we’re using and how they respond. Not all outputs are equal, and not all parts of a prompt are weighted equally.

Iteration 2

Because the output in the first iteration felt diluted, more specifically, summary-like rather than something resembling structured notes, I changed my approach. Instead of concatenating the entire module into one prompt, I fed the model one transcript per API call.

This time, the output was much more detailed and looked more like actual course notes.

This also shows that aside from the system prompt, the content you feed the model in the user prompt matters a lot. Even with generous context windows, thought and experimentation are still needed in how we design these LLM and AI agent pipelines.

Conclusion

• In terms of evaluating the output, I don’t have an evaluation feedback loop in place yet, mainly because we don’t have “model” course notes that represent the ideal output. In practice, we could design these model course notes and run an “LLM-as-a-judge” evaluation loop, asking a smarter LLM to assess the output. For now, we can subjectively evaluate whether the output feels effective or shallow. In the early stages of the project, human judgement is still powerful. An evaluation feedback loop would be ideal once I scale the course notes extraction process.
• Some say prompt engineering is less effective given how smart these models have become. That’s true to some extent, but when you need models to produce a specific kind of output, detailed prompt crafting still matters. This is where trial and error comes in, and consequently why prompt versioning becomes important. Right now, I’m keeping prompt versions in Jupyter notebooks. To be more organized, a good method might be to keep all prompts in a Python file and commit to GitHub regularly.
• Here, I discussed two iterations to keep it simple, but in reality, I iterated many more than 2. Asking a language model to give you the kind of output you want takes time and careful crafting.
• I created a tool to help you estimate your document's token count. It is handy for experimentation. Check it out.

What’s next

In the next post, I’ll discuss the anatomy of a prompt and what I did to get gpt-5-mini to produce the output I wanted.

References

1 AI Engineering by Chip Huyen (O’Reilly). Copyright 2025 Developer Experience Advisory LLC, 978–1–098–16630–4.

2 https://deepmind.google/models/model-cards/

3 https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/start/get-started-with-gemini-3

4 https://platform.claude.com/docs/en/about-claude/models/overview

5 https://developers.openai.com/api/docs/models