SegmentStream's Geo Tests module allows you to conduct experiments that accurately measure the real sales impact of your paid media investments.
This section offers a detailed explanation of the functionality of the module.
Introduction
By comparing the performance of targeted marketing campaigns in a control market where marketing activity remains unchanged, with a similar test market where marketing activities are modified in some way, we can directly observe the impact of these changes by comparing the performance of the two markets.
Currently, SegmentStream supports only one type of treatment - holdout, where campaigns in the test market are completely disabled during the experiment. This allows us to evaluate their incrementality by observing the drop in conversions compared to the control market.
What is the process of running geo-incrementality tests?
The process of running geo-incrementality tests is as follows:
- Determine the scope of the test, decide whether to test an entire channel or a specific set of campaigns.
- Configure test in the SegmentStream admin panel.
- Pause the targeted marketing efforts within the designated test regions.
- Evaluate the results of the test to gain insights.
Getting Started
Before diving into the experiments module:
- Ensure your events tracking is set up - Events tracking.
- Connect at least one data source - Data Sources.
- Define a minimum of one conversion event - Conversions.
- Ensure you have at least two months of conversion data.
Navigation
You'll see a link to the Geo Tests module in the left menu.
Experiment Settings
To launch a geo-test, you need to create an experiment, name it, select the conversion and campaigns that will be used to measure incrementality, and choose an estimated experiment start date. All settings except for the conversion can be changed later.
Market selection
Once the experiment is created, SegmentStream will automatically select the test and control markets, but you can adjust them if necessary.
Market split
Select the geographic level at which the test and control markets will be chosen. Default options include city, country, and region.
Default options rely on IP geolocation, which provides only approximate user locations. Therefore, using more fine-grained geo splits such as “city” based on this data may significantly impact experiment results due to inaccuracies in the approximations.
To address this issue, SegmentStream provides the option to utilize geo data from your CRM for a more reliable market split. The process involves exporting your CRM data into BigQuery, and creating a custom conversion that includes the geo data. The parameters passed from your custom conversions will appear in the selector under the “Custom parameters” label.
For assistance in utilizing CRM data, please contact your SegmentStream manager.
Granularity
If there aren't a significant number of daily conversions available for measuring incrementality, you can select the “Weekly” granularity option.
This choice will impact both the sales cycle setting and how data is visualized in the conversions and costs charts.
Sales cycle
Enter the typical duration for a user to achieve the set goal from their initial interaction with your brand.
Understanding the sales cycle is essential as it determines the period over which the impact of a campaign should be evaluated.
Test and Control markets
The system analyzes trends, seasonality, and volume across various regions, choosing those with the highest correlation. However, it's almost impossible that the volumes will be exactly the same. To address this discrepancy, the synthetic control approach is used, which scales data from control regions in order to more closely align with the test regions' data.
Exclude markets
You can exclude any markets that you don’t want to be included in the experiment. After selecting the excluded markets, the test and control markets will be recalculated.
Minimum detectable effect
The minimum detectable effect (MDE) for an experiment is detected once the test and control markets are selected. The MDE is the smallest change in results that an experiment can reliably detect. If the incrementality is lower than this number, it cannot confidently be attributed to the campaigns within the experiment.
When SegmentStream detects that the MDE exceeds a critical value, a warning will be displayed, indicating that the selected experiment configuration is not recommended.
Result
Once your experiment is launched, the metrics reflecting its progress will be displayed.
Incrementality
This metric presents the calculated incrementality of the campaigns involved in the experiment, as well as the total conversion difference between the test and control markets.
Experiment evaluation
Once the experiment is done, a banner will inform you if the results are strong enough to make clear conclusions and if it's suggested to run the experiment again.
Conversion chart
The conversion chart displays two lines representing conversion trends over time for both test and control markets. It includes the experiment's start and end dates.
Before the experiment starts, SegmentStream's algorithm selects test and control markets in a way that their lines closely align.
Once the experiment begins, we expect the blue line (test markets) to drop below the red line (control markets). This indicates the effectiveness of the campaigns included in the experiment.
Cost chart
The cost chart shows a line depicting the trend in advertising expenditure over time for the chosen test markets. It enables users to verify that the treatment is applied correctly throughout the experiment, indicated by the line remaining close to zero.
