How to Get a Good NLP Baseline When Small Cues Matter

When a few short words decide the label, the baseline starts with reading the text, not with picking a fancier model.

Clinical notes can be very tricky to systematic processing.

Some NLP problems are easy to misread because the decisive evidence is local. A note can look similar at the surface and still mean the opposite thing:

• Reports chest pain x2d, worse on exertion.
• Denies chest pain; shortness of breath improved.
• No active chest pain today.
• Family history of CAD.

If the task is “active chest pain now,” those four lines do not belong in the same bucket. The label turns on small cues: negation, temporality, attribution, and a few abbreviations that must survive preprocessing.

That is the main point of the full post. A good NLP baseline is not just “start with something simple.” It is “make sure the simple thing can still see the meaning-bearing words.”

Start With The Task, Not The Vectorizer

Before you choose bag-of-words, TF-IDF, or a linear model, read a sample of raw texts and decide what the task actually is.

For clinical notes, that means answering questions like:

• What counts as positive?
• What counts as negative?
• What counts as ambiguous?
• Which mistakes matter most in practice?

That sounds basic, but it is where many weak baselines begin. If you have not settled the label policy, a model can look good for the wrong reason. It may be learning note boilerplate, section headers, or superficial word overlap instead of the cue that actually flips the label.

Observation is the text version of exploratory analysis. It is the point where you find out whether the problem is really local, or whether it needs broader context than a sparse baseline can provide.

Cleaning Is Part Of Modeling

Text preprocessing is not a neutral cleanup step. In a task like active symptom detection, it can decide whether the model sees the evidence at all.

Take a fragment like:

[**Name**] reports CP x2d; denies SOB. Family hx CAD.

A useful first-pass cleanup might expand abbreviations and keep the meaning-bearing cues:

<redacted> reports chest pain 2d denies shortness of breath family history cad

That version is not “cleaner” in the abstract. It is better because it preserves the words that matter for the label.

A conservative baseline usually:

• expands clinically meaningful abbreviations
• standardizes de-identification markers
• keeps negation and temporality words
• avoids generic stop-word removal until you have inspected the task
• treats punctuation and section boundaries as potential signal, not automatic noise

The usual stop-word list is a good example of why this matters. In medical text, words like no, not, denies, family, history, and today are not filler. They often carry the decision boundary. Removing them can make the representation look tidy while erasing the distinction you are trying to learn.

Sparse Features Still Earn Their Place

Once the task and cleaning policy are clear, sparse features are still hard to beat as a first baseline.

Single-word counts show you whether important cues are visible at all. Short phrases add back local meaning:

• chest pain
• denies chest pain
• family history
• pain improved
• rule out infection

That matters because a phrase like denies chest pain is not the same evidence as the separate words denies, chest, and pain.

TF-IDF can sit on top of the same vocabulary. It is not a different family of text features so much as a different weighting rule. That helps when boilerplate dominates the corpus, but it does not solve every problem. If the cue depends on temporality, attribution, or longer context, sparse features will still miss cases.

So the question is not whether counts are old-fashioned. The question is whether the task is local enough that counts and short phrases can still show you what is going on.

A Baseline Should Be A Workflow

A good baseline is more than a single model run. It is a decision process you can repeat.

The version I would trust first looks like this:

1. Read a varied sample of raw texts from both labels.
2. Write down the edge cases that keep changing interpretation.
3. Decide what unit is being classified: note, sentence, snippet, or section.
4. Freeze the train, validation, and test split before fitting the vocabulary.
5. Fit the vocabulary on training data only.
6. Inspect the highest-weighted features and the raw false positives and false negatives.
7. Decide whether the next fix is cleaning, labels, phrase features, or a more expressive model.

That sequence matters because it keeps the baseline honest. If the model fails, you want to know whether the failure came from preprocessing, label ambiguity, boilerplate, or the limits of a sparse representation.

Bottom Line

When small cues flip the label, a good baseline is the one that keeps those cues visible.

That usually means conservative cleaning, sparse features you can inspect, and a workflow that starts with reading the text before it starts with fitting the model.

If the baseline cannot preserve negation, temporality, and attribution, a higher score will not teach you much.

Please also check out the full post for the longer version, including the formal expression behind bag-of-words, TF-IDF, and the feature design choices that matter most.