Question

I have the following problem.
I need to calculate the Shrout & Fleiss ICC’s for the situation in which items are judged by a varying number of judges. For example, the competitive nature of an industry is judged for a set of industries, but with a different number of judges per industry. One industry is only judged by 2 judges, whereas the competitive nature of another industry can be judged by up to 12 judges. I have used the ICC (package psych) and icc (package irr) functions for the determination of icc’s, but it is unclear whether they can deal with a varying number of judges. Can anyone help me with this? How to calculate the S&F icc’s when the number of judges is variable? I would prefer to continue using R for this, but if this is better accomplished with another software, I would like to get those suggestions as well.

Thanks, Peter Verbeet

Answer 1

I’ve used hierarchical linear modeling for this. I referred to a Prof. Seltman’s HML notes for R to see how. The trick is getting the variance terms for ICC with variable number of judges.

library(lme4)

# calculate the hierarchical model
m1 = lmer(score ~ (1|item_id) + (1|judge_id), data=d)
print(m1)

# helper function to pull out the variances
xVars <- function(model) {
    exvars = lme4::VarCorr(model)
    vars = c(exvars$item_id[1,1], exvars$judge_id[1,1], attr(exvars,"sc")^2) 
    names(vars) <- c('item var', 'judge var', 'residual var')
    vars
}
# helper function for ICC(k) variations
icck <- function(variances, k=1) {
    icc = variances[1] / (variances[1] + (variances[2] + variances[3]) / k)
    names(icc) = c(paste('ICC', k, sep=''))
    icc
}

# the output you want
icc1 = icck(xVars(m1))
print(icc1)
icc8 = icck(xVars(m1), 8)
print(icc8)