The significance indicates the probability that – taking into account a predefined level of significance – the measured relationship between at least two variables does not occur randomly, but the result can be transferred to the population.
If significance is given, the probability of an error in a hypothesis is below the significance level.
Example:
We assume that the number of shopping cart abandonments is lower when an exit intent layer is applied compared to when no action is taken. For testing this hypothesis, we define a random test group and a control group in an A/B test.
Using significance, the following question can be answered: Is the measured correlation, i.e. fewer shopping cart abandonments when implementing an exit intent measure, transferable to the population, or does the sample represent a random result?
In order to answer the question, a probability of error (p-value) must be defined for the hypothesis. The upper limit of this probability is given by the significance level (α). As a rule, a probability of error with up to 5% deviation, i.e. α=5%, is still considered significant.
With a hypothesis test the p-value can then be calculated. If this is below α=5%, the result is considered significant.
Considering our example, this means that in case of a significant result, the determined correlation of the sample is applicable to the entire sample with a probability of at least 95 percent. With a probability of 5 percent, the measured correlation is random.