Understanding uplift in A/B testing
Let's say we have a(n) (positive) outcome rate which we try to optimize.
Features affects this outcome.
In a test, we find that users that used a specific feature X have a higher positive outcome rate than users that have not used this feature.
What is then the meaning of uplift regarding to the positive outcome rate? Concretely, how would one measure what the uplift in the positive outcome rate would be if all new users started using feature X?
Example:
- Nb users with positive outcome: 10
- Nb users with positive outcome with feature X: 8
- Nb users with positive outcome without feature X: 2
- Nb new users: 5
- Nb new users with feature X and thus a positive outcome: 3
- Nb new users with feature X without positive outcome: 2
What is then the uplift in the positive outcome if all new users would use feature X?