Question

I would like to know how I solve the following problem using higher order functions like ddply, ldply, dlply, and avoid using problematic for loops.

The problem:
I have a .csv file representing a dataset loaded into a data.frame, with each row containing the path to a directory where more information is stored in files. I want to use the directory information in the datas.frame to open the files(“file1.txt”,”file2.txt”) in that directory, merge them, then combine the merged files from each entry in one large dataframe.
something like this:
df =

entryName,dir
1,/home/guest/data/entry1
2,/home/guest/data/entry2
3,/home/guest/data/entry3
4,/home/guest/data/entry4

what I would like to do is apply a function to the dataframe that take the directory,
appends a couple of file names “file1.txt”, “file.txt”, then merges the two files together based off a given field.

for example file1.txt could be:

entry,subEntry,value
1,A,2
1,B,3
1,C,4
1,D,5
1,E,3
1,F,3

for example file2.txt could be:

entry,subEntry,value
1,A,8
1,B,7
1,C,8
1,D,9
1,E,8
1,F,7

the output would look something like this:

entryName,subEntry,valueFromFile1,valueFromFile2
1,A,2,8
1,B,3,7
1,C,4,8
1,D,5,9
1,E,3,8
1,F,3,7
2,A,4,8
2,B,5,9
2,C,6,7
2,D,3,7
2,E,6,8
2,F,5,9

Right now I am using a for loop, but for obvious reasons would like to use a higher order function. Here is what I have so far:

allCombined <- data.frame()
df <- read.csv(file="allDataEntries.csv",header=true) 
numberOfEntries = <- dim(df)[1]

for(i in 1:numberOfEntries){ 
  dir <- df$dir[i]
  file1String <- paste(dir,"/file1.txt",sep='') 
  file2String <- paste(dir,"/file2.txt",sep='')
  file1.df <- read.csv(file=file1String,header=TRUE)
  file2.df <- read.csv(file=file2String,header=TRUE)
  localMerged <- merge(file1.df,file2.df, by="value")
  allCombined <- rbind(allCombined,localMerged) 
} 
#rest of my analysis...

Answer 1

Here is one way to do it. The idea is to create a list with contents of all the files, and then use Reduce to merge them sequentially using the common columns entry and subEntry.

# READ DIRECTORIES, FILES AND ENTRIES
dirs    <- read.csv(file = "allDataEntries.csv", header = TRUE, as.is = TRUE)$dir
files   <- as.vector(outer(dirs, c('file.txt', 'file2.txt'), 'file.path'))
entries <- lapply(files, 'read.csv', header = TRUE)

# APPLY CUSTOM MERGE FUNCTION TO COMBINE ENTRIES
merge_by <- function(x, y){
  merge(x, y, by = c('entry', 'subEntry'))
}
Reduce('merge_by', entries)