From Outsourced to In House: Scaling Market-Based Testing at Thumbtack

Last year, Thumbtack’s Marketing Data Science (DS) team set an ambitious goal: bring all testing fully in-house. Historically, the team had played a critical role in evaluating tests, translating results into strategic direction for the business. However, test design and execution lived elsewhere and were not core to their ownership.

Achieving this shift required more than a new workflow. It called for a cultural transformation within Marketing DS, one that emphasized end-to-end ownership and the ability to deliver clear, high-confidence insights to stakeholders who had long relied on broader, cross-functional input.

But first? We needed testing tools. And for those, we turned to gsynth.

Gsynth is a wordplay on generalized synthetic control method. The modeling package provides causal inference with interactive fixed-effect models. For Marketing DS purposes, gsynth and similar packages can be used to create synthetic controls for Market-Based Tests, and then run Inference tests on this modeled mix to see how a given treatment performs.

Practically speaking: this enables us to design and run market tests quickly and efficiently without futzing about with a well-defined control.

A few DS team members took on the task of adapting basic synthetic control tools for dynamic use cases across the company. Beyond our in-house testing goal, this also allowed the team to establish internal consistency around the mix of tools used, and standardize our approach to validation. One sizable improvement on our initial model was a multi-cell version of gsynth, wherein multiple treatments could be compared to a shared control. This required the model to run AA-tests between variants and iterate through hundreds of market combinations to reach a reasonable outcome, unlocking a capability DS would simply be unable to replicate manually.

Beyond the technical capabilities that gsynth could help us unlock, we also saw opportunity for the DS organization more broadly. By reducing our outsourcing needs, we effectively reduced the footprint of running marketing tests while maintaining the quality of the result. This meant that we were able to run more tests over the course of the year, granted we had the market capacity to do so.

Additionally, gsynth provided a benefit for individual contributors on the DS team, providing them another opportunity to build modeling expertise. By standardizing use of gsynth as our approach to market-based testing, ICs could effectively share understanding, code, and strategy. This enabled them to build collective alignment and feel confident in their ability to model independently.

For more information on gsynth, check out the full description of input factors for the gsynth package here. There’s also a great tutorial here, as well as the originating application in the creator’s thesis. It’s an incredible tool, and what’s more incredible is that it is so readily available for use at companies like ours, that want to level up their in-house testing capabilities.

Interested in joining our team? Check out our open roles at thumbtack.com/careers.