Reliability Analysis

What is reliability analysis?

Reliability analysis is used in several areas, noticeably in social science. The terminology finds its origin in psychometry. We thus define a test made up of questions. Questions are called elements. Those elements are gathered within homogeneous constructs also called factors, measurement scales, latent variables or concepts.

As an illustration, graphical skill might be a factor on which we wish to measure a level on a scale of measurement. The goal of the reliability analysis is to assess the reliability of this scale of measurement, or, in other words, that the construct questions are coherent and measure the same thing. In the case of the graphical skill, a question on mental calculus would downgrade the coherence of the scale.

To the statistician, questions are variables often measured on Likert-type scale (rating answers). Results of a test collected on a group of individuals are gathered in an individuals/variables array. To ensure compatibility with other areas such as quality control where the reliability analysis might also be used, those arrays are labelled observations/variables within XLSTAT.

Methods implemented in XLSTAT are used to estimate the internal consistency of a scale by making sure that results to different questions addressing the same phenomenon are coherent. Moreover, they also measure the reliability between two tests administered to the same individuals at two different times.

The internal analysis, allows to determine which elements of your survey are correlated by providing index related to the internal consistency of the scale but also to identify unnecessary elements and therefore to exclude them.

The split-half reliability analysis measures the equivalence between two parts of a test (parallel forms reliability). This type of analysis is used for two similar sets of items measuring the same thing, using the same instrument and with the same people.

The inter-rater analysis measures reliability by comparing each subject's evaluation variability to the total variability involving all subjects. This analysis has been developed based on the assumption that evaluations may vary according to the rater.

Internal reliability

Cronbach’s alpha

Cronbach's alpha index measures internal consistency, which is, how closely related a set of items are. It is considered to be a measure of scale reliability.

This index is the mathematical equivalent of the average of all correlations between 2 equal portions of the scale. 

Many methodologists recommend a minimum alpha coefficient between 0.65 and 0.8 (or higher in many cases); those which are less than 0.5 are usually unacceptable.

XLSTAT also provides the standardized Alpha coefficient which is equivalent to the reliability that would be obtained if all items values were standardized (centered-reduced variables) before computing Cronbach’s alpha.

Guttman's reliability coefficients (lambda 1-6)

Guttman developed six reliability coefficients: L1 to L6. Four of them (L1, L3, L5 and L6) are used to estimate internal consistency and the last two (L2 and L4) are used for split-half reliabilities:

• L1: Intermediate coefficient used in computing the other lambdas.
• L2: Estimation of the inter-score correlation in the case of parallel measurements. It is more complex than the Cronbach alpha and better represents the true reliability of the test.
• L3: Equivalent to Cronbach's alpha.
• L4: Guttman split-half reliability (See the description below).
• L5: Recommended when a single item strongly covaries with other items whereas those items don’t covary between each other.
• L6: Recommended when inter-item correlations are low compared to the item squared multiple correlations (becomes a better estimator as the number of elements increase).

Split-Half reliability

Spearman-Brown reliability

Another way to calculate the reliability of a scale is to randomly split it into two parts. If the scale is perfectly reliable, we expect the two halves to be perfectly correlated (R = 1). Bad reliability leads to imperfect correlations. Split-Half reliability can be estimated using the Spearman-Brown coefficient. When the two halves have different sizes, a more accurate estimate of this coefficient is used (Horst’s Formula).

Guttman split-half reliability (Split-half model):

Guttman's split-half reliability (L4) is similar to the Spearman-Brown coefficient, but does not consider the reliabilities or variances to be equal in both halves (Tau-equivalence).

Inter-rater reliability

Intra-class Correlation Coefficient (ICC)

Reliability can also be tested using the Inter-rater reliability, with coefficients such as the Intra-class Correlation Coefficient (ICC). This coefficient has several variants but can be generally defined as the part of variance of an observation due to the variability between subjects. ICC values typically vary between 0 and 1.

ICC values are high when there is little variation between the scores given for each question by raters.

Options for setting up a reliability analysis in XLSTAT

Enumerate: Activate this option to find the optimal split-half maximizing the Guttman L4 reliability index for the given input scale. It is carried out by testing every combination in two parts of the initial test.

• Maximum time (s): Activate this option to set a maximum delay in seconds in order to search for the optimal Guttman L4 index (max value).

• Fast  : Use a maximum search algorithm providing an optimal solution in a faster time (preferable when the number of items becomes larger)

Descriptive statistics: Activate this option to display descriptive statistics for the selected variables.

Deleted items statistics: Activate this option to calculate and display statistics on the comparison of each element to the scale composed of others. These statistics include the mean and variance of the scale if the element was deleted, the correlation between the element and the sum of others, Cronbach's alpha if the element was removed from the scale, Guttman’s L6 if the element was removed from the scale and the coefficient of determination (R2) between the deleted element and the scale composed of others.

Correlation matrix: Activate this option to display the correlation matrix that corresponds to the Pearson correlation applied to the observations/items table given as input.

Covariance matrix: Activate this option to display the covariance matrix related to the observations/items table given as input.

Cronbach’s alpha statistics: Activate this option to display the raw and standardized Cronbach’s alphas for the global scale and subscales (only when the "split-half model" reliability is selected).

Guttman’s statistics: Activate this option to display the Guttman’s reliability indices.

• Display the best split-half: Activate this option to display each partition corresponding to an optimal Guttman L4 coefficient.

Split-half model’s statistics: Activate this option to display the reliability indices corresponding to a split-half reliability model (correlation between the two halves of the scale, Spearman-Brown coefficient or Horst coefficient, Guttman split-half reliability coefficient (L4) for the partitions given as input)

Correlation maps: Several visualizations of a correlation matrix are proposed.

• The "blue-red " option allows to represent low correlations with cold colors (blue is used for the correlations that are close to -1) and the high correlations are with hot colors (correlations close to 1 are displayed in red color).

• The "Black and white " option allows to either display in black the positive correlations or in white the negative correlations (the diagonal of 1s is display in grey color), or to display in black the significant correlations, and in white the correlations that are not significantly different from 0.

• The "Patterns " option allows to represent positive correlations by lines that rise from left to right, and the negative correlations by lines that rise from right to left. The higher the absolute value of the correlation, the large the space between the lines.

Results of a reliability analysis in XLSTAT

The correlation matrix and descriptive statistics of scale / item are displayed.

Cronbach's alpha / Guttman’s statistics estimate the reliability of the scale entered as input, while the deleted items statistics provide useful information on the influence of the withdrawal of each element on the overall reliability of the scale.

The internal reliability model allows the computation of the best partition which can be used in the case where a split-half reliability analysis must be considered later.

The correlation map is used to identify possible structures in the correlations and thus to quickly identify elements with interesting correlations.

Example 

An example based on data collected from the Personality Tests website is permanently available on our helpcenter.