Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I have a rather large dataset in text file with about 25 million rows and 200 columns (all of them are numeric). I would like to run some summary statistics and data analysis
(survival analysis) on them.
What is the fastest way to import the data into SAS?
How much memory do I need for my PC in order to run such a large dataset?
I’m not sure that anything is going to be much faster than just reading your dataset in using PROC IMPORT. Specifying your informats and formats in advance might help speed things up a bit, but PROC IMPORT infers these from only the first 20 records by default, so it’s not like it’s going to read your whole dataset to figure out what data types to use. The fact that your columns are all numeric will probably help. The most important thing is to be sure to save the results to a permanent dataset (i.e., specify a library for it) – if you only have to import the data once, it doesn’t really matter if it takes a long time.
One of the nice things about SAS is that it keeps data on disk rather than in memory by default, so the size of your RAM doesn’t really limit the size of your dataset. It might limit what you can do with that dataset, but I don’t know enough about SAS’s internal operations to be able to predict what you’ll have trouble with.