Question

I have a dataset in SPSS that has 100K+ rows and over 100 columns. I want to filter both the rows and columns at the same time into a new SPSS dataset.

I can accomplish this very easily using the subset command in R. For example:

new_data = subset(old_data, select = ColumnA >10, select = c(ColumnA, ColumnC, ColumnZZ))

Even easier would be:

new data  = old_data[old_data$ColumnA >10, c(1, 4, 89)]

where I am passing the column indices instead.

What is the equivalent in SPSS?

I love R, but the read/write and data management speed of SPSS is significantly better.

Answer 1

I am not sure what exactly you are referring to when you write that “the read/write and data management speed of SPSS being significantly better” than R. Your question itself demonstrates how flexible R is at data management! And, a dataset of 100k rows and 100 columns is by no means a large one.

But, to answer your question, perhaps you are looking for something like this. I’m providing a “programmatic” solution, rather than the GUI one, because you’re asking the question on Stack Overflow, where the focus is more on the programming side of things. I’m using a sample data file that can be found here: http://www.ats.ucla.edu/stat/spss/examples/chp/p004.sav

Save that file to your SPSS working directory, open up your SPSS syntax editor, and type the following:

GET FILE='p004.sav'.
SELECT IF (lactatio <= 3).
SAVE OUTFILE= 'mynewdatafile.sav'
  /KEEP currentm previous lactatio.
GET FILE='mynewdatafile.sav'.

More likely, though, you’ll have to go through something like this:

FILE HANDLE directoryPath /NAME='C:\path\to\working\directory\' . 
FILE HANDLE myFile /NAME='directoryPath/p004.sav' . 
GET FILE='myFile'.
SELECT IF (lactatio <= 3).
SAVE OUTFILE= 'directoryPath/mynewdatafile.sav'
  /KEEP currentm previous lactatio.
FILE HANDLE myFile /NAME='directoryPath/mynewdatafile.sav'.
GET FILE='myFile'.

You should now have a new file created that has just three columns, and where no value in the “lactatio” column is greater than 3.

So, the basic steps are:

1. Load the data you want to work with.
2. Subset for all columns from all the cases you’re interested in.
3. Save a new file with only the variables you’re interested in.
4. Load that new file before you proceed.

With R, the basic steps are:

1. Load the data you want to work with.
2. Create an object with your subset of rows and columns (which you know how to do).

Hmm…. I don’t know about you, but I know which method I prefer 😉

If you’re using the right tools with R, you can also directly read in the specific subset you are interested in without first loading the whole dataset if speed really is an issue.