Question

I have a relatively large csv files (1.2gb… large to the 2gb RAM on one of my computers). To every entry in one column I would like to append “1C” so that I can join/merge with another dataframe/db table.

If the file weren’t so large, it would be easy to use read.csv to import to data then use data$symbol <- paste(data$symbol, "1C", sep=""). But now I get the can't allocate vector of size x warning.

Is a manual solution, like scan(), my only option? (I’m a bit afraid of corrupting my data) Thanks!

Answer 1

Using scan isn’t going to help if you can already get your data into R.

Make sure data only has the columns you need to merge, and run gc() before you try your paste command (gc will help if you’re near the margin of your memory limit).

If that fails, look at some of the solutions in this thread.

---

UPDATE:
And if you happen to be using a flavor of *nix, or if you have Rtools installed on windows, you could do this with gawk. If your data are in foo.csv and you want to add the “C1” to the second column, this will create a new file, bar.csv, with “C1” appended to the second column.

compy: /home/josh
> cat foo.csv 
1,one,2,two
3,three,4,four
5,five,6,six

compy: /home/josh
> gawk -F "," '{OFS=","; $2=($2 "1C"); print}' < foo.csv > bar.csv

compy: /home/josh
> cat bar.csv 
1,one1C,2,two
3,three1C,4,four
5,five1C,6,six

This will likely be faster than R and will consume a negligible amount of memory.