Do LLMs Perform Better When They Know the Task?

A small experiment on whether task context helps models understand what they are supposed to solve.

A few days back, I was thinking about something very basic.

Whenever we ask an LLM a question, we usually assume that the model understands what we want it to do. We paste a paragraph and expect it to summarize it. We share two sentences and expect it to classify the relationship between them. We give it a broken sentence and expect it to correct the grammar. We ask a vague question and expect it to figure out whether we want an explanation, a code snippet, a title, a rewrite, a yes/no answer, or something else entirely.

But maybe that assumption is not free. Maybe there is a hidden cost we do not always notice.

Maybe the model is not just answering the question. Maybe, before answering, it is first trying to infer the task itself. Should it classify? Should it rewrite? Should it extract? Should it generate? Should it explain? Should it answer in one word or in a paragraph?

If that is true, then maybe the model is spending some of its effort just figuring out what job it is supposed to do, instead of jumping straight into solving the problem.

I want to be careful here. I am not claiming that we know exactly what is happening inside the model, or which transformer heads are doing what. That would need proper interpretability work. But as a working hypothesis, it is interesting: if we tell the model the task clearly and up front, maybe it does not have to spend as much effort figuring out which mode to use. It can get to the solving part faster.

This is not very different from how we humans work.

If someone gives me a paragraph and says nothing else, I first ask myself, “What am I supposed to do with this?” Should I summarise it? Should I critique it? Should I rewrite it? Should I turn it into code? Should I extract facts from it?

But if someone says, “Fix grammar in the following paragraph,” my mind immediately switches into grammar-correction mode. If someone says, “Write Python code for this,” I switch into coding mode. If someone says, “Turn this into a Medium article,” I switch into writing mode.

The input may be the same, but the task frame changes how we process it.

So the question I wanted to explore was simple: Does the same thing happen with LLMs?

More specifically, if we explicitly tell the model which task it is solving, does it perform better than when we provide only the raw input?

The intuition behind task conditioning

Let’s take a simple example.

Suppose we give a model the following input:

Premise: A man is playing a guitar on stage.
Hypothesis: A person is performing music.

What should the model do?

As humans, many of us can guess that this looks like a textual entailment example. The expected output may be something like entailment. But notice that this is already relying on task inference. The input itself does not explicitly say, “Classify whether the hypothesis follows from the premise.”

Without explicit task guidance, the model may produce responses that seem reasonable but do not match the expected evaluation criteria. It may explain the relationship between the two sentences. It may say, “Yes, the hypothesis is supported.” It may summarise the premise. It may produce a full paragraph. It may even answer correctly at a semantic level, but in the wrong format.

This is one of those hidden problems in LLM evaluation and production systems. The model may understand the content, but if it does not understand the expected task, the final answer can still be wrong for the system.

Now, imagine we instead provide the prompt like this:

Task: Textual Entailment
Input:
Premise: A man is playing a guitar on stage.
Hypothesis: A person is performing music.

This already feels better. The model now has at least a rough idea of what kind of operation we want.

But we can go one step further:

Task: Textual Entailment
Task definition:
Given a premise and a hypothesis, determine whether the hypothesis is entailed by the premise.

Input:
Premise: A man is playing a guitar on stage.
Hypothesis: A person is performing music.

Now the model does not just see the task name. It also gets a sense of what the task actually means.

That small difference is what I wanted to measure.

What we tested

We built a small experiment around this idea. The goal was not to create a new benchmark, but to test a specific question: how much does providing explicit task context affect the output quality of an instruction model?

For every example, we evaluated the same model using four prompt styles.

1. The first was input_only, where the model received only the raw input. This was the baseline.
2. The second was task_name_input, where we added a short task label before the input.
3. The third was task_definition_input, in which we provided both the task label and a full task definition.
4. The fourth was wrong_task_input, where we deliberately gave the model the wrong task name, like “poetry writing,” as a negative control.

The negative control was important. If task context genuinely helps, then a wrong task context should ideally hurt. Otherwise, we may just be measuring the effect of adding more tokens to the prompt.

We used 600 examples from Super-NaturalInstructions, spread evenly across 12 task categories. Each category contributed 50 examples, and each example was tested with all four prompt strategies. So, in total, the model generated 2,400 predictions per run.

Note: We repeated the full generation process 5 times on the same 600 examples. So in total, the model generated:

600 examples × 4 prompt styles × 5 runs = 12,000 predictions

This helped us look not only at the average score, but also at how stable the result was across runs.

We used Qwen3–4B-Instruct via Hugging Face’s router. We evaluated the outputs with several metrics: Normalized Exact Match, Token F1, ROUGE-L, BLEU, and chrF. For this article, I mainly focus on Token F1, because the dataset mixes different types of tasks: classification, extraction, rewriting, correction, and generation. Exact match is useful, but it can be too strict for some of these. Token F1 gives a more flexible cross-task view.

What we found

The overall pattern was quite clear: giving the model the full task definition was the strongest strategy.

Since we ran the full generation process 5 times, the table below reports the average score and standard deviation across runs.

When the model received only the input, the overall Token F1 was 0.178. When we added only the task name, it improved to 0.300. But when we added the full task definition, it jumped to 0.600. That is a large shift, especially because the standard deviation across the 5 runs was very small.

The wrong-task setup is also important here. When we deliberately gave the wrong task context, Token F1 dropped to 0.066. This means the model did not improve just because the prompt became longer. If longer prompts were enough, the wrong-task prompt should also have helped. But it did not. It made the output worse.

So the model seems sensitive to the task frame. The right task context helps. The wrong task context misleads. And no task context leaves the model guessing.

The high-level category-wise view

The overall result is useful, but it can hide category-specific behavior. So we also looked at four deliberately different task types: Textual Entailment, Question Rewriting, Grammar Error Correction, and Data-to-Text.

These four categories are useful because they represent different kinds of model behavior. Textual Entailment is label-sensitive. Question Rewriting is transformation-sensitive. Grammar Error Correction is edit-sensitive. Data-to-Text is generation-heavy and can behave differently from classification or rewriting tasks.

Instead of collapsing these four categories into a single macro average, it is more useful to view them as a matrix.

Token F1 across the four categories

ROUGE-L across the four categories

Textual Entailment is the clearest case where the full task definition mattered. The task name barely helped, but the full definition moved Token F1 from 0.007 to 0.444 and ROUGE-L from 0.009 to 0.628. This suggests the model needed the label semantics, not just the task name.

Question Rewriting also improved strongly. The task name already gave a large jump, and the full definition improved it further. So here, the task name gave a useful routing signal, while the definition added more clarity.

Grammar Error Correction and Data-to-Text behaved differently. In both cases, the shorter task_name_input prompt slightly beat the full definition. The gap was not huge, but it is worth noting. Some tasks may not need a long definition when the task name itself is already clear.

So the category view offers a simple takeaway: task definitions are most useful when the model needs exact task semantics. But more context is not automatically better. The important part is not adding more text, but adding the right task context.

The full category breakdown

The full category view makes the result more nuanced.

The full task definition was the best strategy in 10 out of 12 categories. The task name was best in 2 categories: Data-to-Text and Grammar Error Correction.

So the takeaway is not “always add more instruction.” The better takeaway is: add enough task context to remove ambiguity, but do not assume longer is always better.

Here is the Token F1 breakdown across all 12 categories, averaged across the 5 runs.

The strongest gains came from tasks where the model needed to understand the exact operation being requested. For example, Answerability Classification, Cause-Effect Classification, Dialogue Act Recognition, Textual Entailment, and Coreference Resolution improved sharply only when the full task definition was present.

For rewriting and correction tasks, the task name already gave a strong signal. Question Rewriting improved a lot with just the task name, and the full definition improved it further. Grammar Error Correction was slightly different: the task name performed a little better than the full definition.

Data-to-Text also behaved differently. Here, the task name worked best, although the full definition was close. This is a useful reminder that generation-heavy tasks may not always need a long definition.

So the full breakdown gives a more careful conclusion. Correct task context matters a lot, but the best amount of context depends on the task.

The paired comparison

Since every input was tested with all four prompt styles, we could compare each strategy against input_only on the exact same examples.

This helped check whether the improvement was broad, or whether the average was being driven by a few large wins.

The full task definition improved around 394 out of 600 examples on average, while making only 44 worse. So the gain was broad, not just a few lucky examples pulling up the mean.

The task name also helped, but the effect was smaller. It improved around 265 examples and made around 75 worse.

The negative control tells the other side of the story. The wrong-task setup made around 254 examples worse and improved only around 90. So the model was not just benefiting from a longer prompt. It was benefiting from the correct task frame.

What this suggests

This is where the result becomes interesting to me.

Maybe when we give only the raw input, the model first has to ask, “What am I supposed to do with this?”

Should I classify it? Rewrite it? Extract something? Correct it? Generate something new? Explain it?

Once we give the task name, some of that confusion goes away. Once we provide the task definition, the model receives even clearer direction.

Of course, I am not saying this is exactly how the model works internally. We did not inspect attention heads or activations. But the behavior aligns with intuition: clearer task framing may reduce the effort spent figuring out the task and allow the model to focus more on solving it.

But more context is not always better.

The detailed category breakdown also shows that the optimal amount of task context varies by task. Some tasks need the full definition. Some may only need a short label. Some may get worse if the definition adds too much noise.

So the takeaway is not that we should always make prompts longer.

A better rule is: give the model enough task context to remove ambiguity, but not so much that the instruction itself becomes noise.

This is especially important in production LLM systems, where every extra token has a cost. If a short task label gives most of the benefit, it might be the better choice. If the task is ambiguous or label-sensitive, a full definition might be worth the extra tokens.

How can the industry use this?

This becomes more interesting when we move from benchmarks to real products.

Today, many LLM systems pass the user query directly to the main model along with system prompts, chat history, retrieved context, tools, and formatting rules. The model then has to figure out both things: what the user wants and how to solve it.

But maybe we can separate these two steps.

A lightweight model can first predict the task. It does not need to solve the problem. It only needs to identify whether the user is asking for summarisation, code debugging, extraction, comparison, planning, translation, classification, or something else.

Then the final prompt to the main model can look like this:

<system prompt>
<chat history>
<task>
Likely user intent: code debugging.
Use this as guidance, but prioritize the user query if there is a conflict.
</task>
<user query>
...
</user query>

This gives the main model a starting direction. It does not have to spend as much effort figuring out the task. It can move closer to solving.

This can be useful in customer support, copilots, RAG systems, analytics assistants, coding agents, and workflow automation. A small task-predictor model can be cheaper than the main model, and if it reduces wrong-format answers, retries, or clarification turns, the savings can matter at scale.

But there is one important caution. If the task predictor is wrong, it can hurt the final answer. Our negative-control result showed the same thing: wrong task context made performance worse. So production systems should treat the predicted task as guidance, not absolute truth.

That is why I propose adding the phrase “likely user intent.” It gives the model a useful direction, but does not lock it completely.

So the industry takeaway is not just “write better prompts.” It may be architectural. First, identify the task. Then solve the task.

Humans do this naturally. When we realize we have to write code, we switch into coding mode. When we realize we have to debug, we start looking for failure points.

Maybe LLM systems should work the same way.

What this experiment does not prove

This idea is related to earlier work on instruction-following and Natural Instructions, in which models are trained or evaluated using task descriptions. But the question here is narrower. For the same input and model, what changes when we vary only the task frame during inference?

I want to be careful with the claim.

This experiment does not prove that Qwen3 is literally allocating specific transformer heads to task interpretation and then reallocating them when the task is explicit. That would require a different kind of interpretability study.

It also does not prove that the same pattern will hold for every model, every dataset, or every prompt format. We tested one setup, one model family, one dataset sample, and one task-conditioning style.

It also does not remove the possibility that the model may have seen similar tasks during training. Super-NaturalInstructions is a known dataset, and many instruction-tuned models may have been exposed to similar patterns.

So the claim should be modest.

On this 600-example Super-NaturalInstructions sample, Qwen3 performed substantially better when an explicit task context was provided, especially when the full task definition was included.

That is the main finding. Now it needs more validation.

What I plan to explore next

I see this as the beginning of a research thread, not the final answer.

I want to test this across more models, datasets, and task types. Do larger models need less task framing, or do they also benefit when the task is stated clearly?

I also want to try stronger negative controls. In this experiment, we used the wrong task name. Next, it would be interesting to test wrong task definitions, swapped definitions across tasks, or partially misleading instructions.

And if someone works on interpretability, this becomes even more interesting. Can we see whether explicit task context changes how the model routes the input internally? Does it reduce uncertainty in the output format?

I will continue exploring this, but I would love others to try it too. If you replicate it, challenge it, extend it, or find cases where the opposite happens, please share your findings with me.

The goal is not to prove one prompt format is always better.

The real question is simpler:

How much does clearly telling the model the task change its ability to solve it?

Final thought

We spend so much time talking about bigger models, longer context windows, better RAG pipelines, fine-tuning, agents, and prompting tricks. All of that matters. But this experiment made me pause at a much simpler question.

Did we clearly tell the model what job it is supposed to do?

In our case, that small detail changed the result. The model did better when the task was clear. It did worse when the task context was wrong. So maybe task framing is not just an extra line in the prompt. Maybe it is helping the model enter the right mode before it starts solving.

I will continue exploring this. If you try something similar, extend it, disagree with it, or find cases where the opposite happens, please share your findings with me. I would love to understand where this idea works, where it breaks down, and whether task conditioning can serve as a more systematic way to evaluate instruction-following models.

Because maybe the model was capable all along.

Maybe we were just making it spend too much effort figuring out what we wanted.

**Here is the code **if you want to explore it further. I would love for you to run your own experiments, challenge the results, and share what you find with me.

Also, a small note: Codex was very helpful in this work. Most of the code was written, reviewed, and evaluated with its help.

Keep Exploring!