Question

Being a programmer I occasionally find the need to analyze large amounts of data such as performance logs or memory usage data, and I am always frustrated by how much time it takes me to do something that I expect to be easier.

As an example to put the question in context, let me quickly show you an example from a CSV file I received today (heavily filtered for brevity):

date,time,PS Eden Space used,PS Old Gen Used, PS Perm Gen Used
2011-06-28,00:00:03,45004472,184177208,94048296
2011-06-28,00:00:18,45292232,184177208,94048296

I have about 100,000 data points like this with different variables that I want to plot in a scatter plot in order to look for correlations. Usually the data needs to be processed in some way for presentation purposes (such as converting nanoseconds to milliseconds and rounding fractional values), some columns may need to be added or inverted, or combined (like the date/time columns).

The usual recommendation for this kind of work is R and I have recently made a serious effort to use it, but after a few days of work my experience has been that most tasks that I expect to be simple seem to require many steps and have special cases; solutions are often non-generic (for example, adding a data set to an existing plot). It just seems to be one of those languages that people love because of all the powerful libraries that have accumulated over the years rather than the quality and usefulness of the core language.

Don’t get me wrong, I understand the value of R to people who are using it, it’s just that given how rarely I spend time on this kind of thing I think that I will never become an expert on it, and to a non-expert every single task just becomes too cumbersome.

Microsoft Excel is great in terms of usability but it just isn’t powerful enough to handle large data sets. Also, both R and Excel tend to freeze completely (!) with no way out other than waiting or killing the process if you accidentally make the wrong kind of plot over too much data.

So, stack overflow, can you recommend something that is better suited for me? I’d hate to have to give up and develop my own tool, I have enough projects already. I’d love something interactive that could use hardware acceleration for the plot and/or culling to avoid spending too much time on rendering.

Answer 1

R is a great tool, but I never had to resort to use it. Instead I find python to be more than adequate for my needs when I need to pull data out of huge logs. Python really comes with “batteries included” with built-in support for working with csv-files

The simplest example of reading a CSV file:

import csv
with open('some.csv', 'rb') as f:
    reader = csv.reader(f)
    for row in reader:
        print row

To use another separator, e.g. tab and extract n-th column, use

spamReader = csv.reader(open('spam.csv', 'rb'), delimiter='\t')
for row in spamReader:
   print row[n]

To operate on columns use the built-in list data-type, it’s extremely versatile!

To create beautiful plots I use matplotlib
code

The python tutorial is a great way to get started! If you get stuck, there is always stackoverflow 😉