firebase - 潜在缺陷 - Firebase 在 AAB 测试中的不合理结果

Question

我正在运行 AAB 测试，这意味着测试中有两个对照组，以确保总体分布正确。结果告诉我，复制的对照组比原始组显示出阳性结果。

我在两个应用程序上多次复制了测试，但原始组和复制的对照组之间的结果仍然不同。

这有什么可能的原因吗？

Answer 1

Some percent of online field experiments between identical control groups will show statistical winners. This is called a false positive. Firebase support covers this topic.

Different statistical frameworks have different ways of managing error rate (and false positive rate), eg:

Leanplum

Optimizely

If you experience a false positive multiple times, this is statistically unlikely and usually points to an implementation error or a software error.

Disclaimer: I work at Optimizely.