A Formal Framework for Combining Heterogeneous Scores

Introduction

Weighted averages are everywhere in machine learning systems — ranking candidates, recommending content, blending model outputs with business signals. They look safe and intuitive: pick a few weights, take an average, ship it.

The simplicity is deceptive.

Most scoring systems fail not because the model is weak, but because the scores being averaged are fundamentally incompatible. A model confidence score lives in a narrow band; a behavioral signal spans a much wider one. One score is well-calibrated and linear; another is skewed, noisy, or inverted in meaning. Combine them blindly and the weighted average looks correct on paper but behaves unpredictably in production.

This article makes one claim: weighting is the last step, not the first. Before weights can express importance, the scores themselves must be made comparable in scale, direction, spread, semantics, and independence. Without that groundwork, weights do not control influence — ranges, distributions, and correlations do.

A running example

Throughout this article we will rank job candidates using two signals:

• skill_match — how well the candidate’s skills match the job description (model output, roughly in [0, 1])
• experience — years of relevant prior experience (raw integer, typically 0–25)

The intended formula:

final_score = 0.6 × skill_match + 0.4 × experience

This is the formula we will pull apart, fix, and rebuild.

Why combine scores at all?

A single score rarely captures everything we care about. Quality has multiple dimensions, and each signal measures one of them. The job of a weighted average is not to mix numbers — it is to mix meaning, in a controlled and explainable way, so that improvement on the dimensions we care about most translates into a higher final score.

The trouble is that “mixing meaning” requires the inputs to actually mean comparable things. That is what the rest of this article is about.

The six properties scores must satisfy

For each property below, I’ll state what it means, give a way to check it, and describe the fix.

1. Comparable scale

Property: Every score must live on the same numerical range, so that a unit change in one means roughly what a unit change in another means.

Diagnostic: Plot the histograms side by side. If skill_match ∈ [0, 1] and experience ∈ [0, 25], the experience term will dominate the sum regardless of the weights. The 0.6 / 0.4 wrote down has almost no effect on the actual ranking.

Fix: Normalize, Min-max for bounded, stable distributions; quantile or rank-based scaling for skewed or heavy-tailed ones; z-score when only the relative spread matters. Pick based on the shape of the distribution, not by reflex.

2. Aligned direction

Property: Higher must mean better (or lower must mean better) — consistently, across every score.

Diagnostic: For each score, ask: “If this number goes up, does the candidate get better?” If any answer is “no,” you have a sign problem.

Fix: Invert before normalizing. Common offenders: latency, error rate, distance, cost. A score where lower is better must be flipped (1 - x after normalization, or -x if you are z-scoring).

3. Comparable spread

Property: Beyond having the same range, scores should have similar variance. This is the subtlest of the six and the one most people miss.

Diagnostic: Compute the standard deviation of each normalized score across your dataset. If they differ meaningfully, the high-variance score will dominate ranking even with a small weight.

A quick demonstration. Suppose both scores are normalized to [0, 1] and you weight them 0.5 / 0.5:

If skill_match has std ≈ 0.05 and experience has std ≈ 0.30, the ranking is essentially decided by experience. The weights say "equal" but the variance says otherwise. Ranking listens to spread, not to scale.

Fix: Standardize (z-score) so each score has the same variance, or apply a transformation that flattens the high-variance distribution before normalizing. The choice depends on whether the extra variance is real signal or noise.

4. Linearly meaningful units

Property: Moving from 0.6 to 0.7 should represent the same improvement as moving from 0.8 to 0.9. A weighted sum is a linear operator and assumes linear semantics.

Diagnostic: Bin candidates by score (deciles work well) and plot some external outcome — interview success, conversion, click-through, whatever your ground truth is — against the bin. If the curve is roughly linear, you are fine. If it is flat in the middle and steep at the edges, your score is not linearly meaningful.

Fix: Apply a calibrating transform. Sigmoid for tail-heavy logits. Log for multiplicative scales. Isotonic regression or Platt scaling if you have labels. The goal is not to make the score look prettier — it is to make equal numerical steps correspond to equal real-world steps.

5. Low mutual correlation

Property: The scores should measure different things. This is the property most articles on weighted averages skip, and it is often the one that makes a deployed system underperform.

Diagnostic: Compute the correlation matrix of your normalized scores. If two scores correlate at 0.9, weighting them 0.5 / 0.5 does not give you a balanced view — it gives you roughly a 1.0 weight on one underlying signal that you are double-counting, plus a small residual.

Fix: Three options, in order of effort:

• Drop one of the redundant scores.
• Replace correlated pairs with their principal components, then weight those.
• Orthogonalize: regress one score on the other and use the residual as the second feature.

If your “two signals” are really one signal in disguise, no choice of weights will fix it.

6. Calibration (when scores are probabilities)

Property: If a score is meant to represent a probability — a model’s confidence, a propensity, a likelihood — then 0.7 should mean “happens 70% of the time.” Two probability outputs from different models often disagree on what “0.7” means, and normalizing to [0, 1] does not fix this.

Diagnostic: Reliability plot. Bin predictions by predicted probability, compare to observed frequency in each bin. A well-calibrated score follows the diagonal.

Fix: Platt scaling (fit a logistic regression on the model’s outputs against true labels) or isotonic regression. Calibration is a precondition for treating probabilities from different sources as commensurable.

Choosing the weights

You’ve made the scores comparable. Now you actually have to pick the weights. This is its own problem and deserves to be named, not glossed over.

Three approaches, from least to most data-driven:

Business priors: Stakeholders decide that skill match matters 1.5× more than experience, so weights become 0.6 / 0.4. Fast, transparent, defensible. Works when you have no labels and need to ship.

Grid search against a labeled set: Enumerate weight combinations that sum to 1, score each on a held-out set, pick the best. Works when you have labels and few enough scores that the grid is tractable.

Learn the weights: Treat the final score as the output of a linear model and fit it — logistic regression for binary outcomes, LambdaMART or similar for ranking objectives. The “weights” become learned coefficients. Most flexible, but you lose some of the interpretability that made the weighted average attractive in the first place.

When a weighted average is the wrong tool

A weighted arithmetic mean encodes one assumption: scores are substitutable. Strength on one dimension can compensate for weakness on another. That is often wrong.

• Geometric mean when any zero should kill the result. “The candidate must be good at skills AND experience” — a candidate with skill_match = 0 should not rank highly no matter how much experience they have. The geometric mean enforces this; the arithmetic mean does not.
• Min or threshold gate when every dimension must clear a bar. “Must have ≥ 3 years experience and ≥ 0.5 skill match” — express that as a hard filter, not as weights buried inside an average.
• Pareto ranking when trade-offs are real and you want to surface the frontier rather than collapse it. Useful when the weights themselves are contested or context-dependent.

If you find yourself adding more and more scores to a weighted sum, ask whether the structure of the decision is better expressed as gates plus a smaller average inside each gate.

A note on stability

Everything above assumes the world stands still. It does not.

Min-max parameters, z-score means and standard deviations, calibration curves — all of these are estimated from a sample. When the input distribution drifts in production (new candidate sources, seasonality, an upstream model change), the transformations silently stop doing their job. A score that was centered at 0.5 starts arriving at 0.3, and your weighted average tilts without anyone touching the weights.

Two practical defenses: recompute normalization parameters on a rolling window, and monitor the post-normalization distribution of each score. If the mean or variance moves meaningfully, the weights you chose no longer mean what they meant when you chose them.

Putting it together

Returning to the running example: the pipeline for final = 0.6 × skill_match + 0.4 × experience is not a single line of code. It is roughly:

1. Define what each score means and confirm monotonicity.
2. Align directions (both higher-is-better — already the case here).
3. Normalize to a common range. Min-max for skill_match (already bounded), quantile-rank for experience(heavy-tailed at the top).
4. Equalize variance via z-scoring, or accept the asymmetry deliberately.
5. Check the correlation between the two. If experience strongly predicts skill_match in your data, decide whether to drop, orthogonalize, or accept the double-counting.
6. Calibrate if either score is meant to behave like a probability.
7. Then, and only then, apply the 0.6 / 0.4 weights.
8. Monitor the post-transformation distributions in production and recompute parameters when they drift.

Conclusion

Good weighted averages are engineered, not guessed. The weights are the last and least interesting decision; the work that determines whether the final score behaves well happens upstream, in the comparability, independence, and calibration of the inputs.

If your weighted average is not behaving the way you expect, the fix is almost never to retune the weights. It is to go back to the scores.

Good weighted averages are engineered, not guessed. The quality of the final score is determined long before weights are chosen.