Context engineering… like for real — Delivering Solutions with LLMs (Part 1 of 5)

How vibe coding an MCP server taught me context engineering.

Please note that I’ve written this entire article myself, with only fact checking provided by ChatGPT.

This is part one of a series “Delivering Solutions with LLMs” that I will continue to create as I learn how to deliver completely functional solutions via LLMs.

grok… nice abacus ref, and digging the “abstract background”… Anyone else remember Populus?

One of the issues I’ve faced since I started using LLMs has been to effectively utilize LLMs to solve longer term, deep and wide challenges. Discovering and actively engaging in context engineering has allowed me to develop a workflow to accelerate effectiveness of leveraging LLMs, helping to deliver more stable solutions faster with substantial efficiency gains.

About two months ago, I had used abacus.ai’s DeepAgent to create a signal chat bot for tmdb data requests, but I hit a wall during pre-implementation. While awaiting Google’s automated but long review and approval process to grant a new Google account the ability to create a Voice number, I still wanted to progress the idea. Having set a clear lifespan on my abacus.ai account thanks to the use of an under funded Cash App account, and not having achieved what I wanted before exhausting my abacus.ai credits within 20 days, I canceled my abacus.ai account.

I then came across Gemini CLI in my youtube feed and began using the IDE integration with vscode. Quickly, I became happy with the interactions with and code produced by Gemini.

This article presents a strategy to improve the functionality of my second iteration, the precursor to a new chatbot that has been a great learning experience regarding context engineering, the tmdb MCP server.

ChatGPT… you are one boring image generator.

I was lucky enough that the API of tmdb followed OpenAPI spec, and that a swagger yaml was freely available. I was able to deliver the yaml to gemini and prompted it to build out the MCP tools for all API endpoints, which gemini did quite easily. Gemini had used fastMCP to well form a standards-based MCP interface. It took me about six hours of troubleshooting to get the MCP server functional within Gemini CLI. The reason? I was allowing Gemini CLI (the API client) to infer the use of an http transport for MCP comms when, in fact, the MCP server’s python code was written to offer an stdio transport.

When this challenge was defeated, I began using the tools offered by the MCP server. However, I immediately spotted that Gemini, by default, wasn’t returning well formatted lists of items when I was querying for information.

For example, I’d prompt with:

> search for steal this movie using the
> tmdb tools

The returned information would be the json data payload returned by the API call for which Gemini CLI had coded the decorators/tools for use with fastMCP.

I immediately asked gemini to better format the returned information in a markdown table. Gemini had no problem and immediately returned the table, data which I could now easily review.

However, when I closed the chat without saving then started a fresh chat, the same data formatting issue occurred again. This time, I discovered a pattern that is probably the most important thing to know about working with LLMs… Context Engineering.

Nano Banana, you actually nailed it! I will resige for bread!

Let me break down what I know about working with LLMs so far:

• A human operator uses an API client to send strings over to an API server.
• This API server understands that you have sent strings, and then takes those strings and shoves them down to an “inference layer,” better known, I guess, as the model itself. (I feel like I’m conflating these two things, but… good… go learn more than I know).
• The model then does it’s thing… the thing it does is uses statistical probability to be like “ey yo, next segment of this word is probably ‘ably’ because the first segment of the word is ‘prob’… wait, might be ‘lem’… but what about the rest of the segments around this…” (rinse, repeat)
• The model then returns some strings in the form of words that a human can understand back up the chain and into your eyeballs.

The main point is that, independently… every single time you send a query to the inference layer, there is no knowledge persistence in the model or inference layer. It has “goldfish memory”.

“You are the model now.” — ChatGPT

So, what is context? Context is the thing that keeps track of the memory continuously during your chat engagement over time… over many back-and-forth prompt-and-response exchanges. “Context” in the AI sense is essentially the same definition you already know of the word “context”. Although the model may have items persisted to it (it may respond that “the sky is blue”), the context colors this persisted memory so that it performs better while working with you. For example, I say: “the sky is blue, but today it is gray, you don’t know this because you are a metal box. The sky will be gray for the rest of this conversation.” The model will reply back “Got it, the sky is gray.” And it will “remember” this… but why? How is context maintained?

Context is populated with information as you chat. Believe it or not, every single thing you send over to the inference layer/model lives within the context window (the total “storage box” of context). The data in the context window continues to grow until the data fills up the context window, then items may roll off. The roll off process will delete the oldest context data within the context window and append the newest context data submitted by the user to the “end” of the context window. I believe this function is not managed by the inference layer or the model, but it is managed by the client or the API server!

butwhy.gif. It’s simple. To counteract the “goldfish memory” of the inference layer and model, every single prompt you send, appends all of the context data to the prompt before it arrives to the inference layer/model to be analyzed and responded to.

Additionally, there are some “tricks” you may have already thought of to better manage the content in the context windows… like using another model query to better summarize the content data within the context window, then update the API client or API server (via what I’d theorize is shim logic with a unique user session ID) with this summarized content data.

Nano Banana makin’ that image “crazy! like elephants flying! cats walking on their hind legs! a clown on a unicycle!”

And back to my use case… the tmdb MCP returning unreadable json data after starting a new chat. The answer should be fairly obvious. The context window content that the LLM described to best format json data structures returned by the tmdb API was lost when I started a new chat. So, to keep this concise, I simply asked gemini the following…

> create a file called
> TMDB-MCP_AGENT.md. Populate this
> file with information you can reference
> in the future to better fulfill user
> requests accurately, concisely, most
> efficiently, and most effectively.

The next time I start a fresh chat… all I now do is enter the following:

> Load the contents of
> @TMDB-MCP_AGENT.md and understand
> that this provides guidance to the
> tmdb MCP tool utilization.

Now, whenever I cross another hurdle while working with the tmdb MCP server, I work through it, and then ask gemini to:

> Update the file
> @TMDB-MCP_AGENT.md with
> information you just learned about
> better utilizing the tmdb mcp tools.

tl;dr You should use the LLM to fill it’s own context window with information it produces for itself to do a better job communicating with you.

This isn’t revolutionary; it’s how SYSTEM prompts work. The SYSTEM prompts are prompts that are added to the context window by the API client/server before you interact with the inference layer/model and some of the time cannot be overridden by the user who is interacting with the LLM. What we are doing is adding some USER prompts to the context window that are acting as SYSTEM prompts, advising the LLM how to respond.

It really is this simple.

Now go forth, read The Man With The Shattered World, and get to engineering your context!