How We’re Building Better Experimentation with Synthetic Controls.

By: Thumbtack’s Data Science team

Last year, Thumbtack’s Marketing Data Science (DS) team set an ambitious goal: bring all testing fully in-house (see our first post about this to learn more). Achieving that meant building and adopting a new set of tools and methodologies across the DS organization, including synthetic control models.

To put the new approach into practice, we launched an incrementality experiment expecting a clear signal on which channels were truly driving volume. Instead, the results introduced more ambiguity than answers. As we pressure-tested the model in real time, we uncovered methodological quirks at the exact moment stakeholders needed clarity. It became a powerful lesson in the tension between meeting immediate business needs and investing in long-term analytical infrastructure.

After several weeks of model validation and deeper data investigation, we ultimately reached a high-confidence result for the experiment. Just as importantly, the process surfaced several key lessons that now help guide how we use synthetic control models at Thumbtack today.

Lesson 1: The Dr Pepper Rule

As the mythology goes, Dr Pepper contains 23 flavors, but it’s obvious from the first sip that not every ingredient contributes equally to the final taste. The same is true for synthetic controls. While synthetic control model (SCM) packages like gsynth can accelerate test design, the weighting of underlying control markets can heavily influence the outcome, making it critical for users to evaluate whether the mix is actually reasonable. So, how did we go about influencing the markets in our gsynth package?

Our model used the R best_matches function to pick a list of markets upon which to build our synthetic control. The “best_matches” function finds the best matching control markets for each market in the dataset using dynamic time warping (dtw package). The algorithm simply loops through all viable candidates for each market in a parallel fashion, and then ranks by distance and/or correlation.

However, we discovered that the input variable had a massive impact on the weighted mixture of these inputs. The gsynth inference model will re-calculate weights for each of the control markets to match the pattern of your test market data as best it can. For example, here are the weights that the model output for a revenue oriented model vs. a revenue event oriented model.

Revenue model

Note that the weights can be negative, and can also be very big or very small. It’s critical that users understand the drivers of an SCM mix when assessing the final outcomes.

Multiplying the weights to data from each of the markets, we can see how well gsynth manages to fit the training period for the test markets:

Test vs. Raw Control

Test vs. Linear Weighted Control

We can then use the weighted synthetic control result as a basis of comparison to see how the test does during the test period thereafter.

It’s critical to ensure that your mix of synthetic control markets passes a sniff test when checked against your test. To further pressure test, try adding additional markets, or subtracting a heavily weighted one from your mixture to see how it influences both your visual confirmation and an AA-test.

Lesson 2: Time Is a Flat Circle

In the climax of Slaughterhouse-Five, Billy Pilgrim learns that aliens perceive time not as a linear sequence, but as a complete tapestry of moments existing all at once. Models operate similarly, treating time as an unstructured collection of data points rather than a chronological progression. That distinction becomes especially important when interpreting the “why” behind a model’s results.

To validate the fitness of our synthetic control, we ran an A/A test across the final 4–6 weeks of the training period to confirm the control and test markets remained closely aligned leading into the experiment window. But that approach carries a few implicit assumptions: that the most recent weeks are the most representative, that no non-BAU behavior occurs during the test period, and that nothing meaningfully changes between the end of training and the start of the experiment.

Those assumptions matter more than they may initially appear. External factors like seasonality can affect markets unevenly, and the model itself has no awareness of any gap between the training period and the experiment window. Any divergence during that time is effectively invisible to the model.

Because of this, pressure testing becomes critical. We found it valuable to test whether results held when shifting the A/A window backward or adjusting the surrounding training period. These exercises may create short-term headaches, but they ultimately build far more confidence in the validity of the result.

Bringing testing in-house didn’t give us the tidy answers we expected, but it did give us something more valuable: a deeper understanding of our tools, our data, and the conditions required for trustworthy experimentation. The next step is turning those lessons into muscle memory by codifying them into standards and best practices that make high-quality testing faster, more consistent, and more accessible across Thumbtack.

Interested in joining our team? Check out our open roles at thumbtack.com/careers.