Question

Currently I am loooking for a way to develop an algorithm which is supposed to analyse a large dataset (about 600M records). The records have parameters “calling party”, “called party”, “call duration” and I would like to create a graph of weighted connections among phone users.

The whole dataset consists of similar records – people mostly talk to their friends and don’t dial random numbers but occasionaly a person calls “random” numbers as well. For analysing the records I was thinking about the following logic:

1. create an array of numbers to indicate the which records (row number) have already been scanned.
2. start scanning from the first line and for the first line combination “calling party”, “called party” check for the same combinations in the database
   sum the call durations and divide the result by the sum of all call durations
   add the numbers of summed lines into the array created at the beginning
3. check the array if the next record number has already been summed
   if it has already been summed then skip the record, else perform step 2

I would appreciate if anyone of you suggested any improvement of the logic described above.

p.s. the edges are directed therefore the (calling party, called party) is not equal to (called party, calling party)

Although the fact is not programming related I would like to emphasize that due to law and respect for user privacy all the informations that could possibly reveal the user identity have been hashed before the analysis.

Answer 1

As always with large datasets the more information you have about the distribution of values in them the better you can tailor an algorithm. For example, if you knew that there were only, say, 1000 different telephone numbers to consider you could create a 1000×1000 array into which to write your statistics.

Your first step should be to analyse the distribution(s) of data in your dataset.

In the absence of any further information about your data I’m inclined to suggest that you create a hash table. Read each record in your 600M dataset and calculate a hash address from the concatenation of calling and called numbers. Into the table at that address write the calling and called numbers (you’ll need them later, and bear in mind that the hash is probably irreversible), add 1 to the number of calls and add the duration to the total duration. Repeat 600M times.

Now you have a hash table which contains the data you want.