Question

I have a data frame with 50000 rows and 200 columns. There are duplicate rows in the data and I want to aggregate the data by choosing the row with maximum coefficient of variation among the duplicates using aggregate function in R. With aggregate I can use "mean", "sum" by default but not coefficient variation.

For example

aggregate(data, as.columnname, FUN=mean)

Works fine.

I have a custom function for calculating coefficient of variation but not sure how to use it with aggregate.

co.var <- function(x)
(
 100*sd(x)/mean(x)
)

I have tried

aggregate(data, as.columnname, function (x) max (co.var (x, data[index (x),])

but it is giving an error as object x is not found.

Answer 1

Assuming that I understand your problem, I would suggest using tapply() instead of aggregate() (see ?tapply for more info). However, a minimal working example would be very helpful.

co.var <- function(x) ( 100*sd(x)/mean(x) )

## Data with multiple repeated measurements.
## There are three things (ID 1, 2, 3) that 
## are measured two times, twice each (val1 and val2)
myDF<-data.frame(ID=c(1,2,3,1,2,3),val1=c(20,10,5,25,7,2),
  val2=c(19,9,4,24,4,1))

## Calculate coefficient of variation for each measurement set
myDF$coVar<-apply(myDF[,c("val1","val2")],1,co.var)

## Use tapply() instead of aggregate
mySel<-tapply(seq_len(nrow(myDF)),myDF$ID,function(x){
  curSub<-myDF[x,]
  return(x[which(curSub$coVar==max(curSub$coVar))])
})

## The mySel vector is then the vector of rows that correspond to the
## maximum coefficient of variation for each ID
myDF[mySel,]

EDIT:

There are faster ways, one of which is below. However, with a 40000 by 100 dataset, the above code only took between 16 and 20 seconds on my machine.

# Create a big dataset

myDF <- data.frame(val1 = c(20, 10, 5, 25, 7, 2),
  val2 = c(19, 9, 4, 24, 4, 1))
myDF <- myDF[sample(seq_len(nrow(myDF)), 40000, replace = TRUE), ]
myDF <- cbind(myDF, rep(myDF, 49))
myDF$ID <- sample.int(nrow(myDF)/5, nrow(myDF), replace = TRUE)

# Define a new function to work (slightly) better with large datasets

co.var.df <- function(x) ( 100*apply(x,1,sd)/rowMeans(x) )

# Create two datasets to benchmark the two methods
# (A second method proved slower than the third, hence the naming)

myDF.firstMethod <- myDF
myDF.thirdMethod <- myDF

Time the original method

startTime <- Sys.time()
myDF.firstMethod$coVar <- apply(myDF.firstMethod[,
  grep("val", names(myDF.firstMethod))], 1, co.var)
mySel <- tapply(seq_len(nrow(myDF.firstMethod)),
  myDF.firstMethod$ID, function(x) {
    curSub <- myDF.firstMethod[x, ]
    return(x[which(curSub$coVar == max(curSub$coVar))])
}, simplify = FALSE)
endTime <- Sys.time()

R> endTime-startTime
Time difference of 17.87806 secs

Time second method

startTime3 <- Sys.time()
coVar3<-co.var.df(myDF.thirdMethod[,
  grep("val",names(myDF.thirdMethod))])
mySel3 <- tapply(seq_along(coVar3),
  myDF[, "ID"], function(x) {
    return(x[which(coVar3[x] == max(coVar3[x]))])
}, simplify = FALSE)
endTime3 <- Sys.time()

R> endTime3-startTime3
Time difference of 2.024207 secs

And check to see that we get the same results:

R> all.equal(mySel,mySel3)
[1] TRUE

There is an additional change from the original post, in that the edited code considers that there may be more than one row with the highest CV for a given ID. Therefore, to get the results from the edited code, you must unlist the mySel or mySel3 objects:

myDF.firstMethod[unlist(mySel),]

myDF.thirdMethod[unlist(mySel3),]