Statistical Power for ANOVA / ANCOVA / Repeated measures ANOVA

Ensure optimal power or sample size using power analysis. Power for ANOVA and ANCOVA is available in Excel using the XLSTAT statistical software.

Statistical Power for ANOVA, ANCOVA and Repeated measures ANOVA

XLSTAT-Pro offers tools to apply analysis of variance (ANOVA), repeated measures analysis of variance and analysis of covariance (ANCOVA). XLSTAT-Power estimates the power or calculates the necessary number of observations associated with these models.

When testing a hypothesis using a statistical test, there are several decisions to take:

  • The null hypothesis H0 and the alternative hypothesis Ha.
  • The statistical test to use.
  • The type I error also known as alpha. It occurs when one rejects the null hypothesis when it is true. It is set a priori for each test and is 5%.

The type II error or beta is less studied but is of great importance. In fact, it represents the probability that one does not reject the null hypothesis when it is false. We cannot fix it up front, but based on other parameters of the model we can try to minimize it. The power of a test is calculated as 1-beta and represents the probability that we reject the null hypothesis when it is false.

We therefore wish to maximize the power of the test. The XLSTAT-Power module calculates the power (and beta) when other parameters are known. For a given power, it also allows to calculate the sample size that is necessary to reach that power.

The statistical power calculations are usually done before the experiment is conducted. The main application of power calculations is to estimate the number of observations necessary to properly conduct an experiment. XLSTAT can therefore test:

  • In the case of a one-way ANOVA or more fixed factors and interactions, as well as in the case of ANCOVA:
    • H0: The means of the groups of the tested factor are equal.
    • Ha: At least one of the means is different from another.
  • In the case of repeated measures ANOVA for a within-subjects factor:
    • H0: The means of the groups of the within subjects factor are equal.
    • Ha: At least one of the means is different from another.
  • In the case of repeated measures ANOVA for a between-subjects factor:
    • H0: Les The means of the groups of the between subjects factor are equal.
    • Ha: At least one of the means is different from another.
  • In the case of repeated measures ANOVA for an interaction between a within-subjects factor and a between-subjects factor:
    • H0: The means of the groups of the within-between subjects interaction are equal.
    • Ha: At least one of the means is different from another:

Effect size for ANOVA, ANCOVA and Repeated measures ANOVA

This concept is very important in power calculations. Indeed, Cohen (1988) developed this concept. The effect size is a quantity that will allow calculating the power of a test without entering any parameters but will tell if the effect to be tested is weak or strong. In the context of an ANOVA-type model, conventions of magnitude of the effect size are:

  • f=0.1, the effect is small.
  • f=0.25, the effect is moderate.
  • f=0.4, the effect is strong.

XLSTAT-Power allows you to enter directly the effect size but also allows you to enter parameters of the model that will calculate the effect size. We detail the calculations below:

  • Using variances: We can use the variances of the model to define the size of the effect. With varexplained being the variance explained by the explanatory factors that we wish to test and varerror being the variance of the error or residual variance, we have: f = √varexplained / varerror
  • Using the direct approach: We enter the estimated value of eta² which is the ratio between the explained variance by the studied factor and the total variance of the model. For more details on eta², please refer to Cohen (1988, chap. 8.2). We have: f = √η² / (1 – η²)
  • Using the means of each group (in the case of one-way ANOVA or within subjects repeated measures ANOVA): We select a vector with the averages for each group. It is also possible to have groups of different sizes, in this case, you must also select a vector with different sizes (the standard option assumes that all groups have equal size). We have: f = √Σi (mi - m)² / k / SDintra with mi mean of group i, m mean of the means, k number of groups and SDintra within-group standard deviation.
  • When an ANCOVA is performed, a term has to be added to the model in order to take into account the quantitative predictors. The effect size is then multiplied by f = √1 / (1 – ρ²) where ρ² is the theoretical value of the square multiple correlation coefficient associated to the quantitative predictors.

Once the effect size is defined, power and necessary sample size can be computed.

Calculations for the Statistical Power of ANOVA, ANCOVA and Repeated measures ANOVA

The power of a test is usually obtained by using the associated non-central distribution. For this specific case we will use the Fisher non-central distribution to compute the power.

We first introduce some notations:

  • NbGroup: Number of groups we wish to test.
  • N: sample size.
  • NumeratorDF: Numerator degrees of freedom for the F distribution (see below for more details).
  • NbRep: Number of repetition (measures) for repeated measures ANOVA.
  • ρ: Correlation between measures for repeated measures ANOVA.
  • ε: Geisser-Greenhouse non sphericity correction.
  • NbPred: Number of predictors in an ANCOVA model.

For each method, we give the first and second degrees of freedom and the non-centrality parameter:

  • One-way ANOVA: DF1 = NbGroup – 1; DF2 = N – NbGroup; NCP = f²N
  • ANOVA with fixed effects and interactions: DF1 = NumeratorDF; DF2 = N – NbGroup; NCP = f²N
  • Repeated measures ANOVA within-subjects factor: DF1 = NbRep – 1; DF2 = (N – NbGroup)(NbRep – 1); NCP = f²*N*NbRep*ε / (1 – ρ)
  • Repeated measures ANOVA between-subjects factor: DF1 = NbGroup – 1; DF2 = N – NbGroup; NCP = f²*N*NbRep / [1 + ρ(NbRep – 1)]
  • Repeated measures ANOVA interaction between a within-subject factor and a between-subject factor: DF1 = (NbRep – 1)(NbGroup – 1); DF2 = (N – NbGroup)(NbRep – 1); NCP = f²*N*NbRep*ε / (1 – ρ)
  • ANCOVA: DF1 = NumeratorDF; DF2 = N – NbGroup – NbPred – 1; NCP = f²N

Calculating sample size for ANOVA, ANCOVA and Repeated measures ANOVA taking into account the statistical power

To calculate the number of observations required, XLSTAT uses an algorithm that searches for the root of a function. It is called the Van Wijngaarden-Dekker-Brent algorithm (Brent, 1973). This algorithm is adapted to the case where the derivatives of the function are not known. It tries to find the root of:

power (N) - expected_power

We then obtain the size N such that the test has a power as close as possible to the desired power.

Numerator degrees of freedom for ANOVA, ANCOVA and Repeated measures ANOVA

In the framework of an ANOVA with fixed factor and interactions or an ANCOVA; XLSTAT-Power proposes to enter the number of degrees of freedom for the numerator of the non-central F distribution. This is due to the fact that many different models can be tested and computing numerator degrees of freedom is a simple way to test all kind of models. Practically, the numerator degrees of freedom is equal to the number of group associated to the factor minus one in the case of a fixed factor. When interactions are studied, it is equal to the product of the degrees of freedom associated to each factor included in the interaction.

Suppose we have a 3-factor model, A (2 groups), B (3 groups), C (3 groups), 3 second order interactions A*B, A*C and B*C and one third-order interaction A*B*C We have 3*3*2=18 groups. To test the main effects A, we have: NbGroups=18 and NumeratorDF=(2-1)=1. To test the interactions, eg A*B, we have NbGroups=18 and NumeratorDF=(2-1)(3-1)=2. If you wish to test the third order interaction (A*B*C), we have NbGroups=18 and NumeratorDF=(2-1)(3-1)(3-1)=4. In the case of an ANCOVA, the calculations will be similar.