Why you can’t trust win probability models in D1 college baseball (yet)

Searching for a signal in the noise of run expectancy and win expectancy metrics

While working on year-end player ranking reports for the Ivy League and D1 college baseball, I found myself wishing I could evaluate whether some players do better than others in higher leverage situations. I thought I would just whip up a college win expectancy model and go from there. Instead, I ended up deep in a network of rabbit holes that at least taught me more about what these models mean, and what their uncertainties are. And if you do not want to travel these rabbit holes with me, you can skip to the conclusions that start with Figure 12.

Run Expectancy (RE) and Win Expectancy (WE) are in a class of probabilistic metrics derived from historical inning and game outcomes, respectively (as opposed to direct counting statistics like slugging percentage (SLG) or walks and hits per inning pitched (WHIP)).

RE takes the base_out state and asks, for all plate appearances (PA) on record, what is the average number of runs the batting team scores by the end of the inning (also see this even simpler approach to computing RE). Tom Tango computed RE matrices for different time intervals of Major League Baseball — for example, between 2010–2015, if a batter came to the plate with man on second and one out, the team expected to score an average of 0.664 runs during the remainder of the inning. In 2025, that number is almost the same, at 0.67 (Figure 1).

Figure 1. Fangraphs 2025 MLB Run Expectancy (RE) matrix from Ben Clemens. Note that Fangraphs codes the base state (Runners On), with a ‘dash’ or ‘underscore’ depicting an empty base.

WE (sometimes called Win Probability, WP) takes the base_out_inning_run-differential state and asks, for all games on record, what is the probability that the home team wins. The most common way to compile a WE matrix is to evaluate how often the home team wins whenever a specific base_out_inning_run-differential state is reached — for example, if we have bases loaded with nobody out in the top of the 9th with the home team winning by two, Greg Stoll’s slick MLB WE-finder reports that the home team has a 64% chance of winning (i.e., WE=0.64; Figure 2).

Figure 2. Screenshot (captured June 23, 2025) from Greg Stoll’s WE-finder. With over 120 years of MLB game data, there are 228 instances of this 111_0T9+2 state, revealing a WE=0.64 despite the away team threatening to score. Also note that the 2.32 expected runs (using games from 1903–2024) is a bit lower than the RE=2.69 in 2025 (Figure 1).

You can imagine the attraction of these probabilistic context-dependent statistics. RE evaluates a batter’s (pitcher’s) contribution to scoring (allowing) runs, where a single with the bases loaded and no outs scores a lot more runs than a single with bases empty and two outs. WE measures the impact a batter (pitcher) has on their team’s chance of winning the game. Together, RE and WE help us understand player performance in terms of how high or low leverage the game situation is. For example, if a pitcher gives up hits with no one on and two outs, they might have worse WHIP or TBIP than a pitcher who gives up fewer total hits, but whose hits occur in high leverage situations. Perhaps the pitcher who gives up low-leverage hits is saving their best stuff for high leverage situations and ends up pitching deeper into games — which pitcher would you rather have?

You might also realize that accurate values of RE and WE require a ton of data! RE is not so bad, because there are only 24 unique base_out states. In contrast, when you include extra innings, my D1 College Baseball dataset (more on this later) has 12,907 unique base_out_inning_run-differential states! In Stoll’s WE finders (Figure 2), pitch count is included, further increasing the number of unique states. After subdividing historical play-by-play data into more and more unique states, the question is, do the data have enough instances of each unique state to compile a representative distribution of outcomes with an accurate mean for each state?

As a side note, Tom Tango approaches the problem in a different way. 
First, he demonstrates that you can approximate the RE matrix knowing only 
the runs scored per inning and the home runs hit per inning, averaged over 
the whole season (very clever; https://tinyurl.com/3ptxahjp). Next, he 
simulates games using the average RE matrices to predict runs scored, and 
ultimately, the game winner (https://tangotiger.net/we.html). You will see 
why I choose the probabilistic approach (e.g., Figure 2), rather than this
approach, in my analysis below, because relying only on mean RE values masks
the noisiness of baseball, and underestimates uncertainty on short timescales
(especially in data-limited D1 college baseball)...

In this article, I ask the question whether there are enough D1 College Baseball data to build accurate RE and WE models. Greg Stoll built his WE finder using Retrosheet MLB data from 1903–2024 — that is 193,305 games! I do not have such a ready and large dataset available for D1 baseball, so let’s see what we can do…

In most of our previous SABRtooth articles, we rely on Trackman data. Trackman is the most common optically-enhanced, dual-frequency radar system in D1 baseball, and outputs pitch-by-pitch data for every game in a Trackman-equipped stadium. Trackman data are amazing, and include all sorts of advanced measurements, from exit velocity to pitch break. The problem is, Trackman does not keep track of base runners. BryerCashman from the Iowa Baseball Managers built a WE model from Trackman data anyway, simplifying base-states to keep track only of how many runners are on base, not on which specific bases are occupied. While this creative solution is worth a shot, baserunner location has a huge impact on the results — for example, in MLB, with one out and one runner on, if the runner is on first, RE=0.48, but if the runner is on third, RE=0.86 — a 60% difference (Figure 1).

[embed]Creating a College Baseball Win Probability Model Win probability models have been prominent in the sports world for around a decade now. No matter your sport of choice…medium.com

To my knowledge, there is no free source of college baseball play-by-play data that include baserunner location information (but please, correct me if I am wrong!). A number of baseball data aggregators such as Synergy Baseball, 6–4–3 Charts, and TruMedia do have these data covering the past 4–8 years, but on top of their standard subscription fees, charge an additional $6-$12k/yr for use of their API. So, for the love of the game, I manually assembled Synergy pitch-by-pitch data for the SEC, ACC, AAC, Big12, Big10, Mountain West, and Ivy League conferences spanning 2022–2025. These data contain 10,389 games with 576,621 plate appearances (after removing 670 innings that contained invalid out-states) — just 5.4% of the data that MLB models can use. I will continue to expand these data to other conferences, but let’s see what we can do with what we’ve got.

Figure 3 depicts the number of instances for each base_out state in my Synergy dataset. Usually RE models define the base_out state on the first pitch of a PA (Figure 1, Left). However, especially in college baseball, a lot can happen on the base paths between the first pitch and the last pitch of a PA. By using the base_out state at the end of the plate appearance, I prevent the batter from getting credit for events like wild pitches or stolen bases that happen during their PA but that are not a direct result of the ball they put in play. Therefore, I define a modified Run Expectancy (mRE) matrix where base_out state is set on the last pitch of a PA (Figure 1, Right). The least common base_out state, 001_0 (man on third with no outs; see Table 1 for state coding definitions) has 1,139–2,041 instances, while 83% of mRE base_out states have at least 6,000 instances.

Figure 3. On the left are counts for each of 24 unique base_out states using the traditional approach of using the first pitch of a plate appearance (PA) to compute Run Expectancy (RE). On the right, I depict a modified Run Expectancy (mRE) matrix of counts, where mRE considers the last pitch of a PA when defining the base_out state. Base_out state coding is summarized in Table 1.

Table 1. State Coding.
  Base_Out_Inning_RunDifferential state are written as:
    BBB_O_HI_RD,
      where:
        B  is 0 or 1, 0 meaning no one on that base. The first B is 1st base,
           the second B is 2nd base, and the third B is 3rd base
        O  is the number of outs, 0, 1, or 2.
        H  is the half-inning, T-top or B-bottom.
        I  is the inning, 1, 2, 3, 4, 5, etc..., including extra innings
        RD is the run differential, where +2 means the home team is winning
           by two runs, and -5 means the away team is winning by 5 runs 

Figure 4 compares the D1 College RE and mRE matrices. First, notice that D1 RE values (Figure 4) are significantly higher than MLB RE values (Figure 1), mostly because about 30% more runs are scored in D1 baseball versus MLB. Second, notice that mRE values are systematically lower than RE values (Figure 4), primarily because mRE does not credit batters with enhanced base_out states achieved by advancing base runners during their plate appearance. I’ll focus on mRE from here on out, as I think mRE best isolates the impact of the batter on runs scored.

Figure 4. Means (µ; Top) and standard deviations (1σ; Bottom) for each of 24 unique base_out states using the first pitch of a PA (RE; Left) and the last pitch of a PA (mRE; Right) to compute Run Expectancy. Base_out state coding is summarized in Table 1. In general, D1 College RE values are significantly higher than for MLB (Figure 1), because fewer runs are scored per inning in MLB. RE values are slightly higher than mRE because RE gives batters credit for runners advancing during the PA.

The problem with only presenting mean (µ) mRE matrices (Figure 4, Top Right) is that the user receives no information about the distribution of runs scored (e.g., count, skewness, modality) from a particular base_out state. A quick look at the standard deviation (1σ) of each mRE distribution (Figure 4, Bottom Right) reveals 1σ values an order of magnitude larger than the differences between µmRE values, with the largest 1σ values for base_out states with low counts (Figure 3). Figure 5 depicts the full histograms of the mRE distributions for each base_out state.

Figure 5. Complete histograms of runs scored from each base_out state (Left; RE first pitch of PA. Right; mRE uses last pitch of PA) to the end of the inning. These histograms depict the distributions from which mean and standard deviation are computed in Figure 4. Histograms are colored red, green, and blue for zero, one, and two out states, respectively. To the right of each distribution I list the count (n) and the mean (µ). Notice that some distributions are skewed (assymetric; e.g., 101_0), multimodal (multi-peaked; eg., 011_2), long-tailed (wide range of rare values; e.g., 001_0), or nearly uniform (like a plateau of equally likely values across a wide rangel e.g., 011_1), rendering the exclusive use of means for calculations suspect.

Figures 4 and 5 illustrate two primary concerns for RE analysis. First, in any one situation, µmRE might be quite different than the actual outcome (in some cases, by more than 3 runs 20% of the time!) — in other words, the distributions of runs scored are not narrow or normal (Figure 5). Second, even if your application of mRE involves hundreds of simulations that can thoroughly sample the empirical distribution, the empirical mRE distributions with low counts may have means that do not accurately reflect the true distribution (i.e., it is possible that I do not have enough data to properly characterize the distribution of runs scored in an inning after some baseout states). With these concerns in mind, I ran and experiment that asks the following question: what fraction of viable *after states have mRE distributions distinguishable from their before_state (Figures 6–8)? In other words, if you want to quantify how much a batter or pitcher changes the run expectancy in an inning (using an equation like mRE24 = mRE(after_state) — mRE(before_state) + runsscored; Table 3), are the before and after_state* mRE distributions sufficiently well-characterized and different to yield meaningful results?

To answer this question, first I need to decide how different two mRE distributions need to be to be practically distinguishable. Imeasure the difference in mRE distributions between each viable pair of before and after base_out states using the Earth Mover’s Distance (EMD) metric (Figure 6). Think of each run distribution as a pile of dirt. The EMD is the minimum amount of ‘work’ it would take to move the dirt from the first pile to make it look exactly like the second pile. A small EMD means the piles are already very similar. An EMD of 0.5 runs does not mean the average mRE shifted by 0.5 runs. Instead, the EMD represents the average distance, in runs, that probability must be shifted along the x-axis to make the two mRE distributions identical, accounting for differences in distribution shape, not just mean. Figure 6 depicts a natural threshold at EMD=0.5 runs that allows me to distinguish between base_out state transitions that have a meaningfully large impact on run expectancy (EMD ≥ 0.5), versus those where the difference, while perhaps statistically detectable due to large sample sizes, is too small to reliably influence player evaluation, strategic decisions, or game outcomes (EMD < 0.5). A difference of less than EMD = 0.5 runs is unlikely to change a manager’s strategy or meaningfully impact a player’s evaluation, so we treat any transition below this threshold as effectively indistinguishable.

Figure 6. With 24 unique base_out states (Figures 3 & 4), we have 552 possible transitions. However, more than half of these base_out transitions violate baseball rules, leaving us with 239 viable transitions. For every viable base_out state transition, I calculate the Earth Mover’s Distance (EMD) required to make the two mRE distributions identical. A histogram of these 239 EMD values reveals a Jenks natural break at 0.5 runs, which I take as my distinguishability threshold. In other words, in all future analyses, I will assume that two base_out states are indistinguishable if they have mRE distributions with EMD<0.5.

Let’s discuss the justification for, and the impact of, this distinguishability criterion (Figures 6) with a specific example. Figure 7 (Left) depicts the before = 111_0 to after = 011_2 base_out state transition (i.e., with bases loaded and nobody out, the batter hits into a double play that scores one run). In this case, the large shift in mean mRE dominates the EMD metric, and this base_out state could be reliably modeled even just using the mRE means in Figure 4. Figure 7 (Right) depicts the opposite scenario, where 010_2 and 001_2 not only share very similar mRE means, but also have nearly identical distributions (EMD = 0.054), rendering measurements of change in mRE across this transition unreliable. Figure 7 (Middle) depicts an edge case, where 110_1 and 001_1 share nearly identical mRE means (Figure 4), but their distributions are almost different enough (EMD = 0.437 ~ 0.5) that a thorough Monte Carlo simulation could distinguish between the two states.

Figure 7. An examination of three base_out state transitions spanning the range of distinguishability. Each change-histogram represents the difference between the ‘before’ and ‘after’ base_out state mRE distributions. The larger the mean (|µ|), and the larger the standard deviation (σ) of the change-histogram, the more distinguishable the mRE change is. (Left) One of the most distinguishable base_out state transition goes from 111_0 → 011_2 (with bases loaded and nobody out, the batter hits into a double play that scores one run). This transition has an EMD=2.061 (Figure 6), dominated by the large negative shift in the mean (i.e., the double play drastically reduces run expectancy for the inning). (Middle) In this example, the change-histogram has a mean of zero (µ = 0.01), but the ‘before’ and ‘after’ mRE distributions are different enough in shape that the EMD = 0.437 nearly reaches the 0.5 threshold, making these states almost, but not quite, distinguishable. Note that if you were calculating mRE24 using a table of means (Figure 4), distinguishing the states would be hopeless. (Right) This example of 010_2 → 001_2 has a mean near zero and exceptionally low EMD, making these two base_out states practically indistinguishable (even though a Chi-squared test shows that the two distributions are different at the 95% confidence level).

Now that I have established a threshold of distinguishability (Figure 6), and illustrated some example simulations (Figure 7), it is time to see what fraction of base_out state transitions result in meaningfully different mRE. Figure 8 goes through each of the 24 before base_out states and asks, what fraction of viable after states have distinguishably different mRE distributions. Before states with 0 outs have 50–95% of their viable after states with distinguishable mRE (good news!), but before states with 2 outs have only 0–30% of their viable after states with distinguishable mRE (bad news!).

Figure 8. I compare the distribution of runs scored in an inning from each ‘before’ base_out state to the distribution of runs scored from each viable ‘after’ base_out state using EMD. Each pie chart depicts the fraction of these base_out state transitions that are statistically distinguishable (EMD≥0.5; Figure 6). If a transition is indistinguishable, it means that we cannot hope to quantify the change in mRE caused by a batter or a pitcher using a metric like RE24. The diameter of each pie chart scales with the number of viable ‘after’ states (i.e., how many base_out states can be transitioned to from the ‘before’ base_out state on the graph). The linewidth of the pie chart scales with the count in each ‘before’ base_out state (Figure 3). Some ‘before’ base_out states can be distinguished from viable ‘after’ states 95% of the time (111_0 → BBB_O), while other ‘before’ states can never be distinguished from their viable ‘after’ states (e.g., all but two of the ‘before’ 2-out states).

In summary, I do not think D1 college baseball run expectancy is ready for prime time. With the dataset and approach described above, only 59% of viable base_out state transitions have practically distinguishable before and after run expectancy. Perhaps with some train-test validation experiments, I could refine the EMD distinguishability threshold and open up a few more base_out state transitions to successful modeling. And in the best case scenario, perhaps doubling the size of the dataset could better define some of the sparsest mRE distributions and shift more base_out state transitions to satisfy distinguishability criteria. Nevertheless, if the goal is to use mRE24 to measure the impact that a batter (pitcher) has on creating (allowing) runs over the course of a season, relying on a table of mean mRE values (Figure 4) probably never will yield meaningful results, and even Monte Carlo simulations will struggle to measure accurate change in 41% of offensive events. One way to think about D1 mRE is that it is an effective way of quantifying the largest changes in run expectancy in an inning (like the example in Figure 7 Left, hitting into a double play with bases loaded), but very bad at quantifying small changes, or keeping track of the additive effects of small changes over the course of a game or a season.

Phew, I warned you I was deep in a rabbit hole! It is finally time to tackle win probability. If run expectancy suffers from dataset limitations, imagine the challenge we have with win expectancy, where we have to divide the same data into 12,907 unique states (instead of just 24 for RE)! As a reminder, the goal of Win Expectancy (WE), also called Win Probability (WP), is to measure what percent of the time the home team wins from any base_out_inning_run-differential (BOIRD) state (Table 1). If one can measure the change in WE during any transition between BOIRD states, one can estimate the leverage of the situation, and quantify the impact a batter or pitcher had on the probability of winning the game.

Table 2 shows all the BOIRD states with at least 2000 instances (n). If you recall, n=2,000 is roughly the minimum count we had for the rarest base_out states in the RE analysis (Figure 3), but unfortunately only 0.1% of BOIRD states are this common. 12,548 BOIRD states have n < 300, 11,550 have n < 100, and 10,376 have n < 50. I could stop right here and ask, what hope do we have if 80% of the BOIRD states are characterized by fewer than 50 instances? With all the randomness of a baseball game, there is no way that estimating WE from fewer than 50 instances could yield robust results.

Table 2. Most common base_out_inning_rundiff (BORID) states.

BOIRD          n     win_fraction    µ_run_diff    σ_run_diff
000_0_T1_+0   6028   0.462508295   -0.017750498   6.256598981
000_0_B1_+0   5597   0.635519028    2.770948723   6.280413582
000_1_T1_+0   3980   0.492211055    0.361809045   6.094361349
000_1_B1_+0   3378   0.608940201    2.179692126   6.06039896
000_0_B2_+0   2921   0.608353304    2.137624101   5.545227342
000_0_T2_+0   2871   0.456635319   -0.172065482   5.651832044
000_2_T1_+0   2768   0.521315029    0.749638728   6.056519115
000_2_B1_+0   2281   0.581762385    1.629548444   5.824177091

*See Table 1 for code definitions.

Nevertheless, I will illustrate a few examples before escaping from my rabbit hole. Figure 9 depicts data from fifty BOIRD states with at least 300 instances in the Synergy dataset. Even though this selection of BOIRD states spans the range of win expectancies (0.01–0.97), the end-game run differential (RD) distributions of 80% of the BOIRD states share one standard deviation overlap with each other. In other words, even for the most data dense, best characterized BOIRD states, game-end run differentials are difficult to distinguish. One might say, that’s okay, mean run differential (µRD) varies (Figure 9)… but as we discussed for RE, these RD distributions are so poorly characterized that many of these empirical means are probably inaccurate representations of the BOIRD state.

Figure 9. I select 50 random base_out_inning_run-differential states from the pool of unique BOIRD states with at least 300 instances in my Synergy data set. For each BOIRD state, I plot the mean (µ; colored circle) and one standard deviation (1σ; grey error bar) for the distribution of run differentials from that state to the end of the game. Positive run differentials mean the home team won (Table 1). The xtick labels at the top of the plot list the number of instances of each state, followed by the win fraction (win fractions > 0.5 mean the home team won, Table 1). The size of the colored circles is proportional to the number of instances of that state, and the color of the circles indicates the win fraction. The BOIRD states plotted span nearly the entire range of win fractions (0.01–0.97), but any one BOIRD state has a run differential distribution that has 1σ overlap with as many as 85% of its neighbors.

Despite all my hemming and hawing about not having enough data to properly characterize RE or WE, let’s see what happens if we simulate two example BOIRD state changes. Figure 10 depicts the change from 010_1T3+1 (top of the 3rd inning, one out, man on second, home team winning by 1) to 010_2T3+1 (i.e., the batter was out, and the runner did not advance). Both of these BOIRD states are characterized by over 100 instances in the Synergy dataset (versus n = 3800 to 4600 in the 1903–2024 Retrosheet MLB dataset). You can see that making the second out in the top of the third reduced mRE by 0.55 runs, raised the home team’s chance of winning by 6.6%, and changed the mean game-end run µRD by +0.62. These results seem great, but look at the ΔRD histogram in Figure 10 — one standard deviation is 7.7 runs, and the distribution has long tails and might even be bimodal. Can we really trust such a broad and complex distribution to accurately predict what fraction of the time the home team wins?

Figure 10. I randomly sample (500 times) from ‘before’ (010_1T3+1) and ‘after’ (010_2T3+1) BOIRD state run differential (RD) distributions to simulate changes in win expectancy (WE), µRD, and mRE (using the mRE24 metric). The histogram depicts the distribution of changes in game-end RD after the BOIRD state transition.

Figure 11 shows one more experiment. In this case, I consider a more significant BOIRD state change, but one that is characterized by many fewer instances in the dataset. This time we go from bases loaded, nobody out, in the bottom of the 8th inning with the home team down one run, to man on second with nobody out and the home team up two runs. On the one hand, the results make sense, with the bases clearing double increasing win expectancy by 24.7% and mean run differential by 1.86 after the strongly positive mRE24 event. But how can we trust that just n = 18 instances accurately characterizes the WE or RD for 111_0B8-1? And again, look at the ΔRD histogram, which shows a strongly multimodal and skewed distribution. The mean ΔRD is probably an overestimate, weighed by the population of instances on the right of the histogram where the game became a blowout (Figure 11).

Figure 11. I randomly sample (500 times) from ‘before’ (111_0B8-1) and ‘after’ (010_0B8+2) BOIRD state run differential (RD) distributions to simulate changes in win expectancy (WE), µRD, and mRE (using the mRE24 metric). The histogram depicts the distribution of changes in game-end RD after the BOIRD state transition.

To me, the clear moral of the story is that we need more D1 College Baseball play-by-play data to constrain RE and WE matrices. Except for the largest (<10th and >90th percentile) swings in RE and WE, I just don’t think RE and WE distributions for most state transitions are characterized well enough to evaluate during one plate appearance. I guess the primary question remaining is, if we sum up all these RE and WE values over the course of a game, a season, or a career, does the signal rise above the noise, or are most of the state transitions so poorly characterized that non-random noise accumulates and swamps the signal? I’ll try to wrap up this article by designing a simulation to answer this question.

Let me explain my experimental design (A thru E), so you know exactly where my final result comes from.

A. Simplify the game. I take the Synergy dataset and compute the probability vector of all 667 BOIRD before_states, as well as the transition probability matrix for all 772 BOIRD after_states (after_states include 3-out inning ending states, thus the larger number). In other words, since 2022 in seven D1 conferences, how often does each before_state occur (Figure 3), and how often does each before_state transition to each after_state? Notice that 667x772 is very different from the 12,907 unique BOIRD states I mentioned earlier. To cut to the chase, this model has no hope when using the highly specific BOIRD states defined in Table 1, so I did some binning and redefined the inning and run differential states in Table 2.

Table 2. Simplified inning and run differential states for WE
  Inning States
    Early: Innings 1-6
    Late:  Innings 7-9 (I remove all extra innings)
    T:     Top of inning
    B:     Bottom of inning

   Run Differential (RD) States
    Tied:  Run differential = {0}
    Close: Run differential = {-3, -2, -1, +1, +2, +3}
    Medimum: Run differeential = {-7, -6, -5, -4, +4, +5, +6, +7}
    Blowout: Run differential ≤-8 or ≥+8
    Away: Away team winning (negative RD)
    Home: Home team winning (positive RD)

   Example simplified base_out_inning_rund-differential state
    110_1_LateT_Away_Close: 'Man on first and second, one out, 
      top of an inning late in the game, away team winning by 1-3 runs.'

B. Simulate the league-average player. I simulate each PA by sampling randomly from the probability vector of before_states, and then sampling randomly from the possible after_states that could follow that before_state in the transition matrix. Now I have a random transition, before_state →after_state, and if I repeat this process infinite times, I will reproduce the distributions of states and transitions in the dataset. Next, with a before_state →after_state in hand, I can draw randomly from the histogram of mRE (Figure 5), RD, and WE values in order to calculate mRE24, ΔRD, and WPA (Table 3). If I perform this exact simulation, I will recreate the average D1 hitter, with mRE24, ΔRD, and WPA all approximately zero.

Table 3. Formulas for final simulation.

   mRE24 = mRE(after_state) - mRE(before_state) + runs_scored

if Home Team batting:
   ΔRD  =  (RD(after_state) - RD(before_state))
   WPA  =  (WE(after_state) - WE(before_state))

if Away Team batting
   ΔRD  = -(mRD(after_state) - mRD(before_state))
   WPA  = -(WE(after_state) - WE(before_state))

*mRE is the modified run expectancy for the inning, defined from the last 
        pitch of the PA
*RD  is the end-game run differential, where +2 means the home team won 
        by two runs, and -5 means the away team won by 5 runs
*WE  is the win expectancy (or win probability), where a number 0.5<WE<1.0
        means the home team wins more than the away team, and 0<WE<0.5 means
        the away team wins more than the home team.
*WPA stands for Win Probability Added.

C. Isolate batting skill from clutch. The fun part of this simulation is not just reconstituting the league average hitter from probabilities. My goal is to simulate thousands of hitters who are league average in terms of counting stats, like OBP and SLG, but vary in terms of clutch (and thus mRE24, ΔRD, and WPA). So, my next step is to find counting stats I can compute from before_ and after_ BOIRD states alone (Table 1), instead of using actual play-by-play data. For example, I cannot compute AVG or SLG because singles, walks, hit by pitches, and reached-on-errors all look the same when the only information you have is BOIRD states. So, I calculate mOBP and mISO (Table 4), slightly modified versions of their namesake statistics that we can require every synthetic player to remain league average in. These league averages are mOBP≈0.3966 and mISO≈0.1632.

Table 4. Modified counting statistics.
  mOBP = (HomeRun + Triple + Double + Single + Walk + HitByPitch + Error) / AB
  mISO = (1*Double + 2*Triple + 3*HomeRun) / AB
  AB = All plate appearances except 'Outs classified as Sacrfice'.

D. Create synthetic players. The last step for setting up the simulation is defining three clutch tiers (somewhat arbitrarily) as: Tier 1: Below the 5th percentile of MLB WPA, Tier 2: League Average (WPA≈0), Tier 3: Above the 95th percentile of MLB WPA. Within each clutch tier, I create 500 unique synthetic players, each with their own state transition matrix. I think of these player matrices kind of like APBA Baseball cards. I divide the 772 BOIRD state transitions into 25 bins defined by percentile ranges in WPA (i.e., leverage). Then, I build an optimization to randomly decrease or increase state transitions from each WPA bin until a player’s custom transition matrix creates a theoretical (i.e., at PA=Infinity) WPA that matches their clutch tier, while maintaining league average mOBP and mISO. Table 5 shows a toy example of a single before_state and two possible after_states with drastically different WPA, and how I might modify the transition probabilities in each clutch tier.

Table 5. Toy example of custom transition matrix

Tier 1 (Choke)       Tier 2 (Average)        Tier 3 (Clutch)
    A     B              A     B                 A     B
1  0.97  0.03        1  0.89  0.11           1  0.76  0.24

1 :   Before-State: 111_1_lateT_Home_Close
A :   After-State:  111_3_lateT_Home_Close
            •Inning ending double play; Strongly negative WPA.
B :   After-State:  010_1_LateT_Away_Close
            •Bases clearing double; Strongly positive WPA.
0.XX: Transition Probability. Rows sum to 1. In my actual data, 
        this transition probability matrix is 667x782.

E. Run the simulation. Finally, I now have 500 synthetic player cards (i.e., transition matrices) from each of three clutch tiers, and the simulation begins! I simulate each player-PA by sampling randomly from the probability vector of before_states, and then sampling randomly from the possible after_states that could follow that before_state in the player’s custom transition matrix. For every player-PA, I find the contribution to mOBP and mISO, and draw from the data-derived histograms mRE, RD, and WE to compute contributions to player mRE24, ΔRD, and WPA. The goal here is to ask, how many plate appearances do I need before I can confidently distinguish the least clutch hitters, from league average hitters, from the most clutch hitters, while relying only on simplified BOIRD state, transition, mRE, RD, and WE probabilities derived from the Synergy play-by-play data? This simulation should answer my earlier question about whether, despite the huge variance in mRE, RD, and WE distributions, at some point do I have enough PAs that the noise cancels and I can confidently find the signal?

Figure 12. Simulation of n=500 synthetic players from three clutch tiers, where clutch is measured in units of Win Probability Added per PA. All 1,500 synthetic players have the same mOBP and mISO, set to league average values, so that our study can isolate the effects of clutch hitting from consistently good or bad hitting. Because I do not have good D1 baselines, I set T1 to be equivalent to the bottom 5% of MLB players, T2 is a league average player, and T3 is a top 5% MLB player (in terms of WPA/PA). The table in the title describes the mean and standard deviation of each statistic evolving with PA for 500 T2 league average players. Each subplot shows the kernel density estimate (KDE) of the distribution of WPA/PA for the 500 synthetic players in each tier at each PA checkpoint. In the upper left of each subplot, a small table reports the percent area overlap between each pair of distributions (i.e., the percent of then combined area under both probability distributions that is shared between them). Large overlap means it is more difficult to distinguish the two clutch tiers. With one PA, there is far too much variance in the distributions to measure WPA. It takes 512 plate appearances to confidently (80% of the time) distinguish between the biggest chokers, or the most clutch performer, and a league average guy.

Figure 12 summarizes the results of my simulation, and is a big surprise (at least to me). On the one hand, the simulation shows that trying to interpret WE from one plate appearance is hopeless (not a surprise based on Figures 3–11). At 32 PA (equivalent to about two weeks for a college player), distinguishing whether a batter is clutch or not remains hopeless. With 71% overlap between non-clutch/clutch tiers and the league average tier, you can only be confident about tier assignment for players whose performance falls in the extreme tails (the non-overlapping 29%) of the distribution. Recall that I already am comparing the least possible clutch (T1) and most possible clutch (T3) players to league average, so the signal will never get bigger than what I am simulating, and we are already in the tails! Even at 128 PA (about half a college season), distributions still overlap by 49%, requiring that players perform at the extreme ends of clutch or non-clutch behavior. But, to my surprise, at PA=512 (a shortish season for a major leaguer, or a career for a D1 college player), I now can pretty confidently distinguish the least and most clutch players from league average.

On the one hand, I feel confident that I could measure the clutch hitting ability of a typical D1 college player over a full career (300–500 PA) and determine whether they are a true high-pressure performer. On the other hand, I have even less confidence in the real-time win probability meters that guide in-game decisions and sports betting.

The implication is clear: a model is only as good as its data, and the D1 dataset is simply too sparse for most short-term predictions. My findings show the uncertainty is so large on most state transitions that it renders single-game probabilities statistically meaningless — a sea of noise drowns the signal.

This signal to noise challenge is not just a D1 problem. Anyone using these probabilistic models for high-stakes MLB decisions should perform the same kind of analysis for the much richer MLB dataset. Given the inherent noisiness of baseball, before we trust a model, we first must determine at what timescale — a single plate appearance, a game, or a full season — the signal rises above the noise.

For D1 baseball, that timescale is a career. Only after hundreds of plate appearances does the random noise begin to cancel out, allowing the true measure of a player’s clutch ability to emerge.