By Kai Qi and Momo Meng, Microsoft Experimentation Platform
At Microsoft’s Experimentation Platform (ExP), feature launches are routinely evaluated through A/B tests that can generate hundreds—or even thousands—of metrics for a single experiment.
While this breadth of measurement provides a comprehensive view of feature impact, it also introduces a fundamental statistical challenge: when a scorecard contains hundreds of metric comparisons, how can we tell whether an experiment is actually moving anything, or whether the few statistically significant metrics we observe are simply the result of chance?
This question is more than a statistical curiosity. Misinterpreting random metric movement as a real effect can lead teams to make incorrect product decisions, spend time investigating noise, or overestimate the impact of a feature. Conversely, dismissing genuine signals as noise can cause valuable improvements or regressions to be overlooked.
To address this challenge, ExP uses Treatment Effect Assessment (TEA), a framework that aggregates metric-level statistical evidence into an overall signal for a scorecard slice.
In this post, we describe the methodological challenges of TEA at scale, present empirical findings from large-scale experimentation data, and explain the rationale behind our decision to transition from Fisher’s method to a false discovery rate–based approach.
Why do we need overall treatment effect assessment?
To understand the motivation behind TEA, it is helpful to first consider how scorecards are typically constructed. A scorecard is a collection of metrics used to evaluate the success of a product change or experiment. For consistency and operational efficiency, teams often reuse standardized scorecard templates and shared metric sets across many experiments. Such scorecards may provide broad coverage of a product, even when an experiment targets only a single feature or component. Consequently, not every metric included in a scorecard is expected to respond to a particular treatment.
In practice, experimenters are encouraged to define a diverse set of metrics that capture the impact of a treatment from multiple perspectives, including user engagement, quality, performance, reliability, and business outcomes. As a result, decisions are rarely based on a single metric. Multiple metrics can strengthen confidence in a conclusion by providing converging evidence about the treatment’s impact. However, analyzing a large number of metrics also increases statistical noise and the likelihood of observing significant results purely by chance.
In the age of AI, this challenge becomes even greater. AI can generate custom metrics, slice data in thousands of ways, and run analyses that a human experimenter would never have had the time, or patience, to perform manually. More metrics may sound like an obvious win, but it comes with two catches. First, the more hypotheses we test, the more likely we are to discover something that looks statistically significant simply by chance [2]. This challenge is compounded by the fact that metrics are often related to one another, and those relationships complicate interpretation. Second, not every metric deserves a seat at the decision-making table, and too many metrics that are unrelated to the treatment can bury the signals that actually matter. The question is no longer whether we have enough metrics. It is whether we can trust the conclusions drawn from them.
This is precisely why overall treatment effect assessment is essential. By aggregating evidence across metrics and controlling for false discoveries, TEA helps us avoid mistaking random noise for meaningful signal. However, correcting for multiple comparisons is only part of the problem. As experimentation systems become increasingly agentic, the relevance of the metrics being evaluated matters just as much as the method used to evaluate them. We’ll return to that challenge later. First, let’s focus on assessing overall treatment effects at scale.
What method should we use?
If individual metric p-values tell us what happened to each metric, a TEA p-value attempts to summarize what happened across the scorecard as a whole. There are many methods for combining sets of p-values into an overall TEA p-value. Perhaps the most well-known are Bonferroni [3] and Benjamini-Hochberg [4, 5]. ExP has historically used Fisher’s method [1], but a variety of alternative approaches have also been proposed.
The next question is: which method should we use to combine evidence across metrics? A useful TEA method should provide strong false positive control, remain sensitive to genuine treatment effects, and behave reasonably in the presence of correlated metrics. While correlation-aware approaches exist, they generally require large amounts of historical data or access to raw telemetry to estimate correlation structures, resources that experimentation platforms do not always have available. We therefore focus on p-value aggregation methods that do not require explicit correlation modeling.
Starting from false positive control
To evaluate candidate methods, we analyzed 2,300 scorecards from 14 A/A experiments. Because treatment and control are configured identically in A/A tests, statistically significant results are expected to reflect false positives under the test assumptions. This makes A/A experiments a useful benchmark for validating TEA methods.
On average, we would expect approximately 5% of A/A scorecards to be significant at p < 0.05 and 0.1% at p < 0.001. Table 1 compares the observed False Positive Rates (FPR) across several candidate methods.
| Method | A/A FPR, p < 0.05 | A/A FPR, p < 0.001 |
|---|---|---|
| Benjamini-Hochberg [4, 5] | 2.1% | 0.04% |
| Bonferroni [3] | 1.7% | 0.04% |
| Cauchy [6, 7] | 3.7% | 0.04% |
| Fisher [1] | 22.0% | 12.48% |
| Harmonic [8] | 5.3% | 0.04% |
| Lévy [9] | 1.9% | 0.04% |
| PCCT [10] | 5.1% | 0.04% |
| Storey-Tibshirani [11] | 3.0% | 0.04% |
Most methods produced false positive rates that were reasonably close to the expected levels. Although the observed rates were not exactly 5% and 0.1%, likely because the scorecards were not fully independent, they remained within an acceptable range.
In contrast, Fisher’s method exhibited substantially inflated false positive rates, declaring 22.0% of A/A scorecards significant at the 0.05 level and 12.48% significant at the 0.001 level. The latter is more than 125 times higher than the expected 0.1% false positive rate. Clearly, Fisher’s method can be ruled out from further consideration, but why?
Metrics are not independent
The main problem is that Fisher’s method assumes all p-values are independent, an assumption that rarely holds in practice. Modern experiment scorecards typically contain many related metrics that move together.
For example, a performance scorecard may include page load latency measured at P50, P75, P90, P95, and P99. Similarly, engagement scorecards often contain metrics such as the number of users who used a feature at least once, the number who used it multiple times, and the average usage frequency. In both cases, if one metric moves upward purely by chance, related metrics are likely to move together in the same direction as well.
With Fisher’s method, multiple marginally significant movements can combine into a strongly significant p-value, even though these movements are closely related. Seeing more marginally significant movements in this set after the first one should not add much additional strength to the case against the null hypothesis, but Fisher’s method treats them as additional evidence.
Also, Fisher rewards quantity over strength: when you look at the test statistic for Fisher’s method, -2 Σi ln(pi), you can see that Fisher’s method places substantial weight on the number of moderately significant p-values, rather than the presence of a few extremely significant ones.
For example, seven metric movements each with p = 0.1 have the same contribution to Fisher’s test statistic as one movement with p = 1e-7. But this is the opposite of what we usually want for scorecards. Seven movements each with p = 0.1 can easily occur by chance when a cluster of related metrics all move together. In contrast, a single movement with p = 1e-7 provides strong evidence that the treatment is affecting at least one metric, regardless of what is happening with other metrics in the scorecard.
Ironically, while Fisher’s method has a much higher false positive rate than the other methods, it also failed to detect real movements in some A/B scorecards that had very small p-values.
Method agreement rates
Given the high false positive rate of Fisher’s method, the method historically used for TEA in ExP, we explored alternative approaches. As shown in Table 1, the remaining seven methods all demonstrated reasonable false positive rates on A/A scorecards. The next question was whether they differed meaningfully in other properties, such as statistical power.
As a first step, we compared the methods by measuring their pairwise agreement rates on 44K A/B scorecards when making a binary TEA decision at the 0.05 significance level. In other words, for each pair of methods, we calculated how often they reached the same conclusion about whether a scorecard exhibited an overall treatment effect.
The results revealed a surprisingly high degree of consistency. Excluding Fisher’s method, nearly every pair of methods agreed more than 97% of the time, with some pairs agreeing over 99% of the time. For example, the Lévy and Bonferroni methods reached the same conclusion on 99.6% of scorecards.
| BH | Bonferroni | Cauchy | Harmonic | Lévy | PCCT | Storey-Tibshirani | Fisher | |
|---|---|---|---|---|---|---|---|---|
| BH | 99.3% | 98.4% | 98.9% | 99.6% | 97.3% | 98.5% | 84.5% | |
| Bonferroni | 97.8% | 98.3% | 99.6% | 96.6% | 97.9% | 84.1% | ||
| Cauchy | 99.2% | 98.2% | 98.7% | 98.1% | 85.6% | |||
| Harmonic | 98.8% | 98.1% | 98.3% | 85.2% | ||||
| Lévy | 97.0% | 98.3% | 84.3% | |||||
| PCCT | 97.3% | 85.9% | ||||||
| Storey-Tibshirani | 85.2% |
The key takeaway is that once Fisher’s method is excluded, the choice among the remaining methods becomes much lower-stakes. While there are theoretical differences between these approaches, they arrive at the same practical conclusion for the vast majority of scorecards. As a result, selecting among the non-Fisher candidates is less about avoiding incorrect decisions and more about balancing other important statistical and practical considerations, such as interpretability, power, robustness, and ease of implementation.
Method selection: balancing statistical and practical considerations
Given the comparable statistical performance of several candidate methods, the final selection for ExP required consideration of both theoretical properties and operational constraints.
We selected the Benjamini-Hochberg (BH) procedure as the default method for TEA. This decision was informed by several factors:
- Empirical robustness: In our A/A validation, BH produced false positive rates close to the expected levels, even with the metric correlations commonly observed in experimentation data.
- Interpretability: BH is widely recognized and conceptually straightforward, making it easier to communicate with diverse stakeholders.
- Implementation simplicity: BH requires only ranking and scaling of p-values, avoiding specialized numerical procedures.
- Computational efficiency: BH scales well to the large metric sets typical of production experimentation systems.
The discussion so far has focused on how to aggregate evidence across many metrics while controlling false discoveries. However, the choice of aggregation method is only part of the treatment assessment problem. An equally important question is whether the metrics being aggregated are actually relevant to the experiment being evaluated.
Implications for agentic large-scale experimentation
Great—we now have a method that can control the false discovery rate across a scorecard, even as AI makes it increasingly easy to generate large collections of related metrics. Does that mean experiment evaluation can now be automated based on metric movements and a TEA p-value?
Not quite.
A key consideration for agentic experimentation is that the set of metrics included in the TEA calculation should be relevant to the experiment being evaluated. Otherwise, the resulting assessment may become less sensitive to genuine treatment effects.
Today, ExP reports a BH-adjusted p-value for every scorecard by default. However, the experimentation platform typically has limited context about the experiment itself. Feature flags are often identifiers, and metrics are often little more than names. Without deeper product knowledge, and in some cases due to privacy constraints, the platform cannot reliably determine which metrics are truly related to the feature under test.
As a result, many scorecards contain a mixture of highly relevant metrics and metrics that are unlikely to be affected by the treatment. This is especially common when teams reuse standard scorecard templates across multiple products, surfaces, or feature areas. For example, a small UI change such as adjusting button color, modifying text, or repositioning an element is unlikely to affect unrelated product experiences, even though those metrics may still appear in the same scorecard.
This raises an important question for large-scale, agent-driven experimentation interpretation systems: How many irrelevant metrics can be included before the overall treatment assessment begins to lose sensitivity?
Why irrelevant metrics matter
One perspective comes from the properties of the Benjamini-Hochberg procedure itself. BH is designed to control the false discovery rate, and adding unrelated “A/A-like” metrics does not invalidate that guarantee. The procedure continues to bound the proportion of false discoveries among reported discoveries.
However, this robustness comes with a tradeoff. As more irrelevant metrics are added, BH must adjust for a larger number of tests, making it more difficult for genuinely affected metrics to pass the significance threshold. In other words, while false positive control is maintained, statistical power can decrease.
Another way to view the problem is through the lens of signal dilution. Suppose a scorecard contains a small set of metrics genuinely affected by the treatment and a much larger set of unrelated metrics whose p-values follow the null distribution. As additional unrelated metrics are added, the overall p-value distribution becomes increasingly dominated by noise rather than signal.
While BH continues to control false discoveries, its effectiveness depends on the set of metrics being evaluated. As more metrics that are unlikely to respond to the treatment are included, genuine treatment effects can become harder to detect.
This issue is particularly important in agentic experimentation systems. As AI makes it easier to generate metrics, analyses, and scorecard variants at scale, the challenge is no longer simply correcting for multiple comparisons or accounting for relationships among metrics. It is also ensuring that the metrics included in the assessment are relevant to the experiment being evaluated.
Key takeaways for designing agentic experiment interpretation systems
The practical implication is straightforward: controlling false discoveries is necessary, but it is not sufficient. The quality of the metric set matters just as much as the choice of TEA methodology.
For agentic experiment interpretation systems, this means that metric selection becomes a critical capability. A scorecard composed primarily of metrics that are closely tied to the experiment will generally produce more sensitive, interpretable, and actionable treatment assessments than one crowded with unrelated metrics. While BH is resilient to the inclusion of some irrelevant metrics, an agent that can identify and prioritize experiment-relevant metrics is likely to make better decisions than one that treats every metric equally.
References
[1] R. A. Fisher, “Questions and answers #14,” The American Statistician, 2(5), 30–31, 1948.
[2] K. L. Sainani, “The Problem of Multiple Testing,” PM&R, vol. 1, no. 12, pp. 1098–1103, 2009.
[3] O. J. Dunn, “Multiple Comparisons Among Means,” Journal of the American Statistical Association, 56(293), 52–64, 1961.
[4] Y. Benjamini and Y. Hochberg, “Controlling the false discovery rate: a practical and powerful approach to multiple testing,” Journal of the Royal Statistical Society, Series B, 57(1), 289–300, 1995.
[5] Y. Benjamini, “Discovering the false discovery rate,” Journal of the Royal Statistical Society, Series B, 72(4), 405–416, 2010.
[6] Y. Liu and X. Xie, “Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures,” Journal of the American Statistical Association, 115(529), 393–402, 2020.
[7] Y. Fang et al., “Heavy-tailed distribution for combining dependent p-values with asymptotic robustness,” Statistica Sinica, 33, 1115–1142, 2023.
[8] D. J. Wilson, “The harmonic mean p-value for combining dependent tests,” Proceedings of the National Academy of Sciences, 116(4), 1195–1200, 2019.
[9] D. J. Wilson, “The Lévy Combination Test,” arXiv:2105.01501 (opens in new tab).
[10] Y. Ouyang et al., “Effective Positive Cauchy Combination Test,” arXiv:2410.10345 (opens in new tab).
[11] J. D. Storey and R. Tibshirani, “Statistical significance for genomewide studies,” Proceedings of the National Academy of Sciences, 100(16), 9440–9445, 2003.
Artificial intelligence 
Data platforms and analytics